[00:11:32] 3MediaWiki-extensions-Flow, operations: Non-Flow pages can be moved to the Topic namespace, and cannot be moved back - https://phabricator.wikimedia.org/T87396#990498 (10Nihiltres) There might be a use-case for moving stuff to/from the Topic namespace, but until one's established, and pages can be moved back, I... [00:12:29] (03PS1) 10Jdlrobson: Enable the wikidata mobile experiments on beta labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186319 [01:00:22] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: puppet fail [01:19:12] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [02:11:39] !log l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s) [02:11:43] !log LocalisationUpdate completed (1.25wmf14) at 2015-01-23 02:11:43+00:00 [02:11:49] Logged the message, Master [02:11:53] Logged the message, Master [02:19:41] !log l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s) [02:19:45] !log LocalisationUpdate completed (1.25wmf15) at 2015-01-23 02:19:45+00:00 [02:19:48] Logged the message, Master [02:19:52] Logged the message, Master [02:37:24] 3operations, Wikimedia-DNS: wikibook.org should be hosted by WMF and should redirect to wikibooks.org - https://phabricator.wikimedia.org/T87375#990546 (10Dzahn) we don't own the domain name: Registrant Name:Glynipher Co Registrant Organization:Glynipher Co Registrant Street: 522 Lopez Jaena st Registrant City:... [02:39:15] (03CR) 10Dzahn: "the idea behind doing cross-box NAT was that we would switch DNS of gerrit.wikimedia.or to point to misc-web in order to proxy it and the" [puppet] - 10https://gerrit.wikimedia.org/r/185340 (https://phabricator.wikimedia.org/T84713) (owner: 10Dzahn) [04:03:46] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Jan 23 04:03:45 UTC 2015 (duration 3m 44s) [04:03:52] Logged the message, Master [04:30:23] (03PS1) 10Krinkle: conf: Clean up highlight.php and fix http-redirect bug [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186327 [04:30:31] (03CR) 10jenkins-bot: [V: 04-1] conf: Clean up highlight.php and fix http-redirect bug [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186327 (owner: 10Krinkle) [04:33:40] (03PS2) 10Krinkle: conf: Clean up highlight.php and fix http-redirect bug [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186327 [04:35:01] (03CR) 10Krinkle: Replace easter egg by a more explaining message (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/110926 (https://bugzilla.wikimedia.org/60741) (owner: 10Hashar) [05:07:11] PROBLEM - git.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:13:01] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 59741 bytes in 0.069 second response time [05:32:51] PROBLEM - ElasticSearch health check for shards on logstash1002 is CRITICAL: CRITICAL - elasticsearch http://10.64.32.137:9200/_cluster/health error while fetching: Request timed out. [05:32:51] PROBLEM - ElasticSearch health check for shards on logstash1001 is CRITICAL: CRITICAL - elasticsearch http://10.64.32.138:9200/_cluster/health error while fetching: Request timed out. [05:32:51] PROBLEM - ElasticSearch health check for shards on logstash1003 is CRITICAL: CRITICAL - elasticsearch http://10.64.32.136:9200/_cluster/health error while fetching: Request timed out. [05:56:52] PROBLEM - puppet last run on mw1010 is CRITICAL: CRITICAL: Puppet has 1 failures [06:14:41] RECOVERY - puppet last run on mw1010 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:28:31] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:32] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:42] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:21] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 2 failures [06:32:21] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:21] PROBLEM - check_puppetrun on lutetium is CRITICAL: CRITICAL: Puppet has 1 failures [06:40:21] PROBLEM - check_puppetrun on lutetium is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:21] PROBLEM - check_puppetrun on lutetium is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:51] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:46:11] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:46:12] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:46:31] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:46:31] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:50:21] RECOVERY - check_puppetrun on lutetium is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [07:18:41] PROBLEM - puppet last run on lvs2002 is CRITICAL: CRITICAL: Puppet has 5 failures [07:18:41] PROBLEM - puppetmaster backend https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8141: HTTP/1.1 500 Internal Server Error [07:18:41] PROBLEM - puppet last run on cp4012 is CRITICAL: CRITICAL: Puppet has 1 failures [07:18:51] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: Puppet has 23 failures [07:18:52] PROBLEM - puppet last run on pollux is CRITICAL: CRITICAL: puppet fail [07:18:52] PROBLEM - puppet last run on elastic1009 is CRITICAL: CRITICAL: puppet fail [07:18:56] this is going to be fun [07:19:00] puppetmaster died [07:19:01] PROBLEM - puppet last run on cp4002 is CRITICAL: CRITICAL: Puppet has 1 failures [07:19:11] PROBLEM - puppet last run on db1027 is CRITICAL: CRITICAL: Puppet has 12 failures [07:19:12] PROBLEM - puppet last run on analytics1021 is CRITICAL: CRITICAL: Puppet has 19 failures [07:19:12] PROBLEM - puppet last run on labsdb1005 is CRITICAL: CRITICAL: Puppet has 10 failures [07:19:12] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: puppet fail [07:19:12] PROBLEM - puppet last run on mw1103 is CRITICAL: CRITICAL: Puppet has 27 failures [07:19:21] PROBLEM - puppet last run on mc1015 is CRITICAL: CRITICAL: Puppet has 13 failures [07:19:21] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: Puppet has 10 failures [07:19:21] PROBLEM - puppet last run on mw1128 is CRITICAL: CRITICAL: Puppet has 27 failures [07:19:22] PROBLEM - puppet last run on db1068 is CRITICAL: CRITICAL: Puppet has 21 failures [07:19:31] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: Puppet has 54 failures [07:19:31] PROBLEM - puppet last run on es2006 is CRITICAL: CRITICAL: Puppet has 12 failures [07:19:31] PROBLEM - puppet last run on db2010 is CRITICAL: CRITICAL: Puppet has 11 failures [07:19:31] PROBLEM - puppet last run on db2017 is CRITICAL: CRITICAL: Puppet has 12 failures [07:19:31] PROBLEM - puppet last run on bast2001 is CRITICAL: CRITICAL: Puppet has 13 failures [07:19:32] PROBLEM - puppet last run on cp3011 is CRITICAL: CRITICAL: Puppet has 8 failures [07:19:32] PROBLEM - puppet last run on hydrogen is CRITICAL: CRITICAL: Puppet has 17 failures [07:19:32] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: Puppet has 38 failures [07:19:32] PROBLEM - puppet last run on mw1047 is CRITICAL: CRITICAL: puppet fail [07:19:33] PROBLEM - puppet last run on elastic1029 is CRITICAL: CRITICAL: Puppet has 14 failures [07:19:33] PROBLEM - puppet last run on labsdb1002 is CRITICAL: CRITICAL: puppet fail [07:19:34] PROBLEM - puppet last run on virt1009 is CRITICAL: CRITICAL: Puppet has 14 failures [07:19:41] PROBLEM - puppet last run on es1004 is CRITICAL: CRITICAL: Puppet has 13 failures [07:19:41] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Puppet has 11 failures [07:19:41] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: puppet fail [07:19:41] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: Puppet has 17 failures [07:19:41] PROBLEM - puppet last run on amssq50 is CRITICAL: CRITICAL: Puppet has 19 failures [07:19:41] PROBLEM - puppet last run on analytics1031 is CRITICAL: CRITICAL: Puppet has 11 failures [07:19:42] PROBLEM - puppet last run on tmh1001 is CRITICAL: CRITICAL: Puppet has 19 failures [07:19:42] PROBLEM - puppet last run on db1009 is CRITICAL: CRITICAL: Puppet has 10 failures [07:20:01] PROBLEM - puppet last run on lvs2005 is CRITICAL: CRITICAL: Puppet has 21 failures [07:20:02] PROBLEM - puppet last run on mw1078 is CRITICAL: CRITICAL: puppet fail [07:20:02] PROBLEM - puppet last run on mw1230 is CRITICAL: CRITICAL: puppet fail [07:20:02] PROBLEM - puppet last run on es1003 is CRITICAL: CRITICAL: puppet fail [07:20:02] PROBLEM - puppet last run on cp4011 is CRITICAL: CRITICAL: Puppet has 23 failures [07:20:02] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: Puppet has 18 failures [07:20:03] PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: Puppet has 22 failures [07:20:03] PROBLEM - puppet last run on mw1095 is CRITICAL: CRITICAL: puppet fail [07:20:04] PROBLEM - puppet last run on mw1199 is CRITICAL: CRITICAL: Puppet has 38 failures [07:20:04] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: puppet fail [07:20:05] PROBLEM - puppet last run on cp1064 is CRITICAL: CRITICAL: Puppet has 13 failures [07:20:05] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: Puppet has 33 failures [07:20:06] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: Puppet has 39 failures [07:20:11] PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: puppet fail [07:20:11] PROBLEM - puppet last run on db1058 is CRITICAL: CRITICAL: puppet fail [07:20:11] PROBLEM - puppet last run on mw1194 is CRITICAL: CRITICAL: Puppet has 42 failures [07:20:11] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: puppet fail [07:20:12] PROBLEM - puppet last run on mw1070 is CRITICAL: CRITICAL: Puppet has 71 failures [07:20:12] PROBLEM - puppet last run on mw1058 is CRITICAL: CRITICAL: puppet fail [07:20:12] PROBLEM - puppet last run on mw1075 is CRITICAL: CRITICAL: puppet fail [07:20:31] PROBLEM - puppet last run on mw1094 is CRITICAL: CRITICAL: puppet fail [07:20:32] PROBLEM - puppet last run on tmh1002 is CRITICAL: CRITICAL: puppet fail [07:20:32] PROBLEM - puppet last run on ms-be1005 is CRITICAL: CRITICAL: puppet fail [07:20:41] PROBLEM - puppet last run on db1019 is CRITICAL: CRITICAL: Puppet has 15 failures [07:20:41] PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: puppet fail [07:20:41] PROBLEM - puppet last run on ms-be2009 is CRITICAL: CRITICAL: Puppet has 17 failures [07:20:42] PROBLEM - puppet last run on ms-be1014 is CRITICAL: CRITICAL: puppet fail [07:20:42] PROBLEM - puppet last run on ms-be2010 is CRITICAL: CRITICAL: puppet fail [07:20:42] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: Puppet has 27 failures [07:20:51] PROBLEM - puppet last run on mw1179 is CRITICAL: CRITICAL: Puppet has 36 failures [07:20:52] PROBLEM - puppet last run on labsdb1007 is CRITICAL: CRITICAL: Puppet has 15 failures [07:20:52] PROBLEM - puppet last run on mw1191 is CRITICAL: CRITICAL: puppet fail [07:20:52] PROBLEM - puppet last run on analytics1029 is CRITICAL: CRITICAL: Puppet has 11 failures [07:20:52] PROBLEM - puppet last run on mw1101 is CRITICAL: CRITICAL: Puppet has 68 failures [07:20:52] !log restarted puppetmaster and apache2 on palladium. [07:21:01] Logged the message, Master [07:21:02] PROBLEM - puppet last run on mw1127 is CRITICAL: CRITICAL: puppet fail [07:21:11] PROBLEM - puppet last run on erbium is CRITICAL: CRITICAL: Puppet has 47 failures [07:21:11] RECOVERY - puppetmaster backend https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.047 second response time [07:21:12] PROBLEM - puppet last run on lvs4001 is CRITICAL: CRITICAL: puppet fail [07:21:12] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: Puppet has 18 failures [07:21:12] PROBLEM - puppet last run on wtp1014 is CRITICAL: CRITICAL: Puppet has 26 failures [07:21:12] PROBLEM - puppet last run on cp1057 is CRITICAL: CRITICAL: Puppet has 16 failures [07:21:12] PROBLEM - puppet last run on mw1196 is CRITICAL: CRITICAL: puppet fail [07:21:12] PROBLEM - puppet last run on mw1169 is CRITICAL: CRITICAL: Puppet has 74 failures [07:21:21] PROBLEM - puppet last run on es1005 is CRITICAL: CRITICAL: Puppet has 10 failures [07:21:21] PROBLEM - puppet last run on mw1083 is CRITICAL: CRITICAL: Puppet has 45 failures [07:21:21] PROBLEM - puppet last run on analytics1024 is CRITICAL: CRITICAL: Puppet has 14 failures [07:21:22] PROBLEM - puppet last run on mw1214 is CRITICAL: CRITICAL: Puppet has 74 failures [07:21:22] PROBLEM - puppet last run on cp1051 is CRITICAL: CRITICAL: puppet fail [07:21:22] PROBLEM - puppet last run on praseodymium is CRITICAL: CRITICAL: puppet fail [07:21:32] PROBLEM - puppet last run on db1041 is CRITICAL: CRITICAL: puppet fail [07:21:32] PROBLEM - puppet last run on amssq58 is CRITICAL: CRITICAL: puppet fail [07:21:41] PROBLEM - puppet last run on db1024 is CRITICAL: CRITICAL: puppet fail [07:21:42] PROBLEM - puppet last run on elastic1016 is CRITICAL: CRITICAL: Puppet has 12 failures [07:21:42] PROBLEM - puppet last run on mw1246 is CRITICAL: CRITICAL: puppet fail [07:21:42] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: puppet fail [07:21:51] PROBLEM - puppet last run on mw1257 is CRITICAL: CRITICAL: puppet fail [07:21:51] PROBLEM - puppet last run on ms-be2015 is CRITICAL: CRITICAL: puppet fail [07:21:52] PROBLEM - puppet last run on db1010 is CRITICAL: CRITICAL: puppet fail [07:21:52] PROBLEM - puppet last run on amslvs4 is CRITICAL: CRITICAL: puppet fail [07:21:52] PROBLEM - puppet last run on cp1069 is CRITICAL: CRITICAL: puppet fail [07:21:52] PROBLEM - puppet last run on mw1013 is CRITICAL: CRITICAL: Puppet has 58 failures [07:22:01] PROBLEM - puppet last run on mw1232 is CRITICAL: CRITICAL: puppet fail [07:22:02] PROBLEM - puppet last run on mw1136 is CRITICAL: CRITICAL: Puppet has 32 failures [07:22:02] PROBLEM - puppet last run on elastic1026 is CRITICAL: CRITICAL: puppet fail [07:22:02] PROBLEM - puppet last run on mw1132 is CRITICAL: CRITICAL: puppet fail [07:22:11] PROBLEM - puppet last run on mw1234 is CRITICAL: CRITICAL: Puppet has 29 failures [07:22:11] PROBLEM - puppet last run on mw1035 is CRITICAL: CRITICAL: puppet fail [07:22:11] PROBLEM - puppet last run on stat1001 is CRITICAL: CRITICAL: Puppet has 30 failures [07:22:12] PROBLEM - puppet last run on mw1036 is CRITICAL: CRITICAL: Puppet has 30 failures [07:22:12] PROBLEM - puppet last run on mw1182 is CRITICAL: CRITICAL: Puppet has 80 failures [07:22:21] PROBLEM - puppet last run on mc1010 is CRITICAL: CRITICAL: puppet fail [07:22:21] PROBLEM - puppet last run on mw1244 is CRITICAL: CRITICAL: puppet fail [07:22:21] PROBLEM - puppet last run on mw1138 is CRITICAL: CRITICAL: Puppet has 36 failures [07:22:21] PROBLEM - puppet last run on mw1096 is CRITICAL: CRITICAL: Puppet has 70 failures [07:22:22] PROBLEM - puppet last run on cp4017 is CRITICAL: CRITICAL: Puppet has 27 failures [07:22:22] PROBLEM - puppet last run on analytics1019 is CRITICAL: CRITICAL: Puppet has 24 failures [07:22:31] PROBLEM - puppet last run on zirconium is CRITICAL: CRITICAL: Puppet has 47 failures [07:22:31] PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: puppet fail [07:22:31] PROBLEM - puppet last run on cp1043 is CRITICAL: CRITICAL: Puppet has 14 failures [07:22:31] PROBLEM - puppet last run on mw1184 is CRITICAL: CRITICAL: Puppet has 33 failures [07:22:31] PROBLEM - puppet last run on mw1161 is CRITICAL: CRITICAL: puppet fail [07:22:31] PROBLEM - puppet last run on db2041 is CRITICAL: CRITICAL: Puppet has 9 failures [07:22:32] PROBLEM - puppet last run on es1006 is CRITICAL: CRITICAL: Puppet has 14 failures [07:22:42] PROBLEM - puppet last run on db2035 is CRITICAL: CRITICAL: puppet fail [07:22:42] PROBLEM - puppet last run on cp4013 is CRITICAL: CRITICAL: Puppet has 19 failures [07:22:42] PROBLEM - puppet last run on amssq57 is CRITICAL: CRITICAL: Puppet has 27 failures [07:22:42] PROBLEM - puppet last run on elastic1028 is CRITICAL: CRITICAL: puppet fail [07:22:51] PROBLEM - puppet last run on labsdb1001 is CRITICAL: CRITICAL: Puppet has 11 failures [07:22:52] PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: puppet fail [07:22:52] PROBLEM - puppet last run on analytics1039 is CRITICAL: CRITICAL: puppet fail [07:23:01] PROBLEM - puppet last run on db1007 is CRITICAL: CRITICAL: Puppet has 7 failures [07:23:01] PROBLEM - puppet last run on mc1008 is CRITICAL: CRITICAL: Puppet has 12 failures [07:23:01] PROBLEM - puppet last run on virt1002 is CRITICAL: CRITICAL: Puppet has 13 failures [07:23:02] PROBLEM - puppet last run on mw1028 is CRITICAL: CRITICAL: puppet fail [07:23:02] PROBLEM - puppet last run on es2003 is CRITICAL: CRITICAL: puppet fail [07:23:02] PROBLEM - puppet last run on ocg1001 is CRITICAL: CRITICAL: puppet fail [07:23:02] PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: Puppet has 32 failures [07:23:02] PROBLEM - puppet last run on ms-be2002 is CRITICAL: CRITICAL: puppet fail [07:23:11] PROBLEM - puppet last run on analytics1036 is CRITICAL: CRITICAL: puppet fail [07:23:11] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: puppet fail [07:23:11] PROBLEM - puppet last run on mw1089 is CRITICAL: CRITICAL: puppet fail [07:23:11] PROBLEM - puppet last run on cp1065 is CRITICAL: CRITICAL: Puppet has 25 failures [07:23:12] PROBLEM - puppet last run on virt1012 is CRITICAL: CRITICAL: Puppet has 8 failures [07:23:12] PROBLEM - puppet last run on ytterbium is CRITICAL: CRITICAL: Puppet has 22 failures [07:23:13] PROBLEM - puppet last run on cp3022 is CRITICAL: CRITICAL: puppet fail [07:23:13] PROBLEM - puppet last run on wtp1024 is CRITICAL: CRITICAL: Puppet has 24 failures [07:23:13] PROBLEM - puppet last run on cp1059 is CRITICAL: CRITICAL: puppet fail [07:23:13] PROBLEM - puppet last run on es1009 is CRITICAL: CRITICAL: puppet fail [07:23:14] PROBLEM - puppet last run on ms-fe1003 is CRITICAL: CRITICAL: puppet fail [07:23:21] PROBLEM - puppet last run on mw1147 is CRITICAL: CRITICAL: puppet fail [07:23:21] PROBLEM - puppet last run on mw1080 is CRITICAL: CRITICAL: puppet fail [07:23:21] PROBLEM - puppet last run on mw1134 is CRITICAL: CRITICAL: puppet fail [07:23:22] PROBLEM - puppet last run on mw1109 is CRITICAL: CRITICAL: Puppet has 34 failures [07:23:22] PROBLEM - puppet last run on mc1004 is CRITICAL: CRITICAL: puppet fail [07:23:31] these should stop shortly [07:23:31] PROBLEM - puppet last run on mw1124 is CRITICAL: CRITICAL: Puppet has 77 failures [07:23:32] PROBLEM - puppet last run on mw1130 is CRITICAL: CRITICAL: Puppet has 28 failures [07:23:32] PROBLEM - puppet last run on mw1245 is CRITICAL: CRITICAL: puppet fail [07:23:41] PROBLEM - puppet last run on mw1221 is CRITICAL: CRITICAL: Puppet has 27 failures [07:23:41] PROBLEM - puppet last run on mw1240 is CRITICAL: CRITICAL: puppet fail [07:23:42] PROBLEM - puppet last run on mw1040 is CRITICAL: CRITICAL: Puppet has 37 failures [07:23:43] PROBLEM - puppet last run on analytics1015 is CRITICAL: CRITICAL: Puppet has 14 failures [07:23:51] PROBLEM - puppet last run on mw1178 is CRITICAL: CRITICAL: puppet fail [07:23:51] PROBLEM - puppet last run on mw1252 is CRITICAL: CRITICAL: Puppet has 83 failures [07:23:52] PROBLEM - puppet last run on cp4007 is CRITICAL: CRITICAL: Puppet has 15 failures [07:23:52] PROBLEM - puppet last run on vanadium is CRITICAL: CRITICAL: Puppet has 12 failures [07:23:52] PROBLEM - puppet last run on amssq43 is CRITICAL: CRITICAL: puppet fail [07:23:52] PROBLEM - puppet last run on cp3013 is CRITICAL: CRITICAL: Puppet has 33 failures [07:24:02] PROBLEM - puppet last run on mc1011 is CRITICAL: CRITICAL: puppet fail [07:35:42] RECOVERY - puppet last run on cp4002 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:36:11] RECOVERY - puppet last run on elastic1029 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:36:12] RECOVERY - puppet last run on analytics1031 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [07:36:31] RECOVERY - puppet last run on lvs2002 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:36:32] RECOVERY - puppet last run on cp4012 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:36:42] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [07:36:42] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:36:52] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [07:36:52] RECOVERY - puppet last run on amssq37 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [07:36:52] RECOVERY - puppet last run on analytics1021 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [07:37:01] RECOVERY - puppet last run on labsdb1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:37:01] RECOVERY - puppet last run on mw1103 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:37:01] RECOVERY - puppet last run on mc1015 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [07:37:02] RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [07:37:02] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:37:02] RECOVERY - puppet last run on mw1128 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:37:11] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [07:37:12] RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:37:12] RECOVERY - puppet last run on es2006 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:37:12] RECOVERY - puppet last run on db2010 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:37:12] RECOVERY - puppet last run on db2017 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [07:37:21] RECOVERY - puppet last run on hydrogen is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [07:37:21] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [07:37:21] RECOVERY - puppet last run on bast2001 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [07:37:21] RECOVERY - puppet last run on cp3011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:37:22] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:37:22] RECOVERY - puppet last run on virt1009 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [07:37:22] RECOVERY - puppet last run on es1004 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:37:22] RECOVERY - puppet last run on tmh1001 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:37:22] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [07:37:23] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:37:24] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:37:24] RECOVERY - puppet last run on cp1066 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [07:37:31] RECOVERY - puppet last run on elastic1010 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:37:41] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [07:37:41] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [07:37:51] RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [07:37:51] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [07:37:52] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:37:52] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [07:37:52] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:37:52] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [07:37:52] RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [07:37:52] RECOVERY - puppet last run on db1058 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [07:38:01] RECOVERY - puppet last run on elastic1009 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:38:01] RECOVERY - puppet last run on mw1058 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [07:38:01] RECOVERY - puppet last run on pollux is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [07:38:02] RECOVERY - puppet last run on db1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:02] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:38:02] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [07:38:02] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [07:38:02] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [07:38:03] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:38:12] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [07:38:21] RECOVERY - puppet last run on ms-be1005 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [07:38:22] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [07:38:22] RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [07:38:31] RECOVERY - puppet last run on ms-be2009 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:38:32] RECOVERY - puppet last run on ms-be2010 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [07:38:32] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [07:38:32] RECOVERY - puppet last run on labsdb1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:41] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:38:41] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:38:41] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:38:42] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:38:42] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [07:38:42] RECOVERY - puppet last run on labsdb1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:42] RECOVERY - puppet last run on analytics1029 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [07:38:42] RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [07:38:42] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:43] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:38:52] RECOVERY - puppet last run on erbium is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [07:39:01] RECOVERY - puppet last run on mw1230 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:39:01] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:39:01] RECOVERY - puppet last run on es1003 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:39:01] RECOVERY - puppet last run on lvs2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:39:02] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:39:02] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [07:39:02] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:39:02] RECOVERY - puppet last run on cp1057 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:39:03] RECOVERY - puppet last run on cp4011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:39:11] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:39:11] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:39:11] RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [07:39:12] RECOVERY - puppet last run on es1005 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [07:39:12] RECOVERY - puppet last run on mw1184 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:39:13] RECOVERY - puppet last run on mw1083 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:39:13] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:39:13] RECOVERY - puppet last run on analytics1024 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [07:39:13] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:39:13] RECOVERY - puppet last run on mw1070 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [07:39:14] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:39:14] RECOVERY - puppet last run on praseodymium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:39:21] RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [07:39:22] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [07:39:22] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:39:31] RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [07:39:31] RECOVERY - puppet last run on elastic1016 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [07:39:31] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:39:32] RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:39:41] RECOVERY - puppet last run on tmh1002 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [07:39:41] RECOVERY - puppet last run on mc1008 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:39:51] RECOVERY - puppet last run on ms-be2015 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:39:51] RECOVERY - puppet last run on analytics1036 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:39:51] RECOVERY - puppet last run on cp1069 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [07:39:51] RECOVERY - puppet last run on cp1065 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:39:52] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [07:39:52] RECOVERY - puppet last run on mw1013 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [07:39:52] RECOVERY - puppet last run on mw1232 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [07:40:01] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:40:02] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:40:02] RECOVERY - puppet last run on elastic1026 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [07:40:02] RECOVERY - puppet last run on mw1234 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:40:02] RECOVERY - puppet last run on mw1127 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:40:02] RECOVERY - puppet last run on stat1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:40:11] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:40:11] RECOVERY - puppet last run on mw1036 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [07:40:12] RECOVERY - puppet last run on mc1010 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [07:40:12] RECOVERY - puppet last run on mw1096 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [07:40:12] RECOVERY - puppet last run on mw1130 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [07:40:21] RECOVERY - puppet last run on mw1245 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [07:40:21] RECOVERY - puppet last run on analytics1019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:40:21] RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:40:22] RECOVERY - puppet last run on cp4017 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:40:22] RECOVERY - puppet last run on zirconium is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:40:22] RECOVERY - puppet last run on cp1043 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:40:23] RECOVERY - puppet last run on cp1051 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:40:31] RECOVERY - puppet last run on db2041 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:40:31] RECOVERY - puppet last run on es1006 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [07:40:31] RECOVERY - puppet last run on mw1252 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [07:40:32] RECOVERY - puppet last run on elastic1028 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [07:40:32] RECOVERY - puppet last run on cp4013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:40:41] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [07:40:42] RECOVERY - puppet last run on labsdb1001 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [07:40:42] RECOVERY - puppet last run on mw1246 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [07:40:51] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:40:51] RECOVERY - puppet last run on analytics1039 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [07:40:51] RECOVERY - puppet last run on db1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:40:52] RECOVERY - puppet last run on virt1002 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:40:52] RECOVERY - puppet last run on mw1257 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [07:41:01] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:41:01] RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:41:02] RECOVERY - puppet last run on virt1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:41:02] RECOVERY - puppet last run on ytterbium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:41:11] RECOVERY - puppet last run on es1009 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:41:11] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [07:41:11] RECOVERY - puppet last run on mw1147 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [07:41:11] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [07:41:12] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:41:21] RECOVERY - puppet last run on mw1109 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:41:21] RECOVERY - puppet last run on mw1124 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [07:41:21] RECOVERY - puppet last run on mw1244 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:41:22] RECOVERY - puppet last run on mw1138 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:41:31] RECOVERY - puppet last run on mw1221 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [07:41:32] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [07:41:32] RECOVERY - puppet last run on mw1040 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [07:41:32] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [07:41:41] RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:41:41] RECOVERY - puppet last run on vanadium is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [07:41:42] RECOVERY - puppet last run on db2035 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:41:42] RECOVERY - puppet last run on cp4007 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [07:41:52] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:41:52] RECOVERY - puppet last run on mc1011 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [07:42:11] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [07:42:11] RECOVERY - puppet last run on ocg1001 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:42:12] RECOVERY - puppet last run on es2003 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [07:42:12] RECOVERY - puppet last run on mw1089 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:42:12] RECOVERY - puppet last run on ms-be2002 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [07:42:12] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [07:42:21] RECOVERY - puppet last run on cp1059 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:42:21] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:42:22] RECOVERY - puppet last run on mw1080 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:42:22] RECOVERY - puppet last run on mw1134 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [07:42:31] RECOVERY - puppet last run on mc1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:42:41] RECOVERY - puppet last run on mw1240 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:42:51] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:43:01] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:43:01] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:25:06] (03CR) 10Ebrahim: [C: 031] Allow sysops/botadmins to grant/revoke 'confirmed' user group on fawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186230 (https://phabricator.wikimedia.org/T87348) (owner: 10Calak) [09:28:52] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 0 below the confidence bounds [09:43:12] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 0 below the confidence bounds [09:55:11] PROBLEM - check_mysql on db1008 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 896 [10:00:11] RECOVERY - check_mysql on db1008 is OK: Uptime: 584997 Threads: 9 Questions: 1038088 Slow queries: 3897 Opens: 9369 Flush tables: 2 Open tables: 64 Queries per second avg: 1.774 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [10:43:41] RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected [13:10:01] PROBLEM - puppet last run on labnet1001 is CRITICAL: CRITICAL: puppet fail [13:28:51] RECOVERY - puppet last run on labnet1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:36:22] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 525 bytes in 0.006 second response time [13:48:22] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.008 second response time [15:37:00] 3WMF-NDA-Requests, operations, WMF-Legal: Add multichill to WMF-NDA group - https://phabricator.wikimedia.org/T87097#991056 (10Qgil) #WMF-Legal, #Operations, could someone verify that @multichill (Maarten Dammers) has an NDA in place? [15:42:52] 3WMF-NDA-Requests, operations, WMF-Legal: Grant WMF-NDA access to Stas in Phabricator - https://phabricator.wikimedia.org/T85170#991059 (10Qgil) #WMF-legal, could you verify that an NDA signed by @Smalyshev (Stas Malyshev) is in place, please? Also, there must be a simpler and quicker way to verify that a WMF e... [15:43:14] 3WMF-NDA-Requests, operations, WMF-Legal: Grant WMF-NDA access to Stas in Phabricator - https://phabricator.wikimedia.org/T85170#991062 (10Qgil) p:5Triage>3Normal [15:46:21] 3WMF-NDA-Requests, operations: Grant Nikerabbit access to WMF-NDA group - https://phabricator.wikimedia.org/T86632#991063 (10Qgil) @Nikerabbit's case seems obvious to me, but I am not entitled to grant access to WMF-NDA to anybody. Can someone in #Operations grant Niklas access, please? [17:11:58] 3WMF-NDA-Requests, operations, WMF-Legal: Grant WMF-NDA access to Stas in Phabricator - https://phabricator.wikimedia.org/T85170#991127 (10Dzahn) >>! In T85170#991059, @Qgil wrote: > Also, there must be a simpler and quicker way to verify that a WMF employee has signed an NDA. could that be legalpad.wm.org (in... [17:12:32] (03CR) 10John F. Lewis: [C: 031] "Patch is good. Removing wikibook.org might not be worth it." [puppet] - 10https://gerrit.wikimedia.org/r/185474 (https://phabricator.wikimedia.org/T87039) (owner: 10Glaisher) [17:14:38] !log logstash elasticsearch cluster split brained; logstash1002 thinks it is a lone master [17:14:46] Logged the message, Master [17:17:21] !log logstash1002 elasticsearch rejoined cluster after restart [17:17:31] RECOVERY - ElasticSearch health check for shards on logstash1002 is OK: OK - elasticsearch status production-logstash-eqiad: status: yellow, number_of_nodes: 3, unassigned_shards: 12, timed_out: False, active_primary_shards: 55, cluster_name: production-logstash-eqiad, relocating_shards: 0, active_shards: 149, initializing_shards: 4, number_of_data_nodes: 3 [17:17:31] Logged the message, Master [17:17:32] RECOVERY - ElasticSearch health check for shards on logstash1001 is OK: OK - elasticsearch status production-logstash-eqiad: status: yellow, number_of_nodes: 3, unassigned_shards: 12, timed_out: False, active_primary_shards: 55, cluster_name: production-logstash-eqiad, relocating_shards: 0, active_shards: 149, initializing_shards: 4, number_of_data_nodes: 3 [17:17:51] RECOVERY - ElasticSearch health check for shards on logstash1003 is OK: OK - elasticsearch status production-logstash-eqiad: status: yellow, number_of_nodes: 3, unassigned_shards: 12, timed_out: False, active_primary_shards: 55, cluster_name: production-logstash-eqiad, relocating_shards: 0, active_shards: 149, initializing_shards: 4, number_of_data_nodes: 3 [17:33:49] (03PS1) 10Milimetric: Use new db hostnames [puppet] - 10https://gerrit.wikimedia.org/r/186356 [17:42:01] (03PS3) 10Giuseppe Lavagetto: snapshot: unify node declarations, use role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/185152 [17:46:07] (03PS17) 10Andrew Bogott: Add class and role for Openstack Horizon [puppet] - 10https://gerrit.wikimedia.org/r/170340 [17:49:51] PROBLEM - puppet last run on labnet1001 is CRITICAL: CRITICAL: Puppet has 1 failures [17:54:41] 3Ops-Access-Requests, Continuous-Integration: Make sure relevant RelEng people have access to gallium (Chris M, Dan, Mukunda, Zeljko) - https://phabricator.wikimedia.org/T85936#991202 (10RobH) a:5JohnLewis>3RobH I'll handle this later today. Ops is in break out sprints, which include a phabricator group (wh... [17:55:52] RECOVERY - puppet last run on labnet1001 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [17:59:50] !log apt-get clean, free some diskspace on antimony [18:00:05] Logged the message, Master [18:00:32] PROBLEM - puppet last run on cp3019 is CRITICAL: CRITICAL: Puppet has 1 failures [18:00:33] PROBLEM - puppet last run on cp3007 is CRITICAL: CRITICAL: Puppet has 1 failures [18:01:03] PROBLEM - puppet last run on ms-be3004 is CRITICAL: CRITICAL: Puppet has 1 failures [18:01:05] (03PS1) 10RobH: cleaning out holmium entries [puppet] - 10https://gerrit.wikimedia.org/r/186357 [18:01:33] PROBLEM - puppet last run on amssq45 is CRITICAL: CRITICAL: Puppet has 1 failures [18:01:33] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: Puppet has 1 failures [18:02:12] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 7.14% of data above the critical threshold [500.0] [18:02:22] PROBLEM - puppet last run on wtp1024 is CRITICAL: CRITICAL: Puppet has 1 failures [18:03:14] (03PS3) 10BBlack: disable nginx udplog for SSL clusters [puppet] - 10https://gerrit.wikimedia.org/r/186257 [18:03:17] (03PS2) 10BBlack: use /run/ directly for pidfile under systemd [puppet] - 10https://gerrit.wikimedia.org/r/186258 [18:04:05] (03CR) 10BBlack: "Apparently "service nginx reload" doesn't work when operating through the symlink in the nginx config file, but this fixes it. Don't ask " [puppet] - 10https://gerrit.wikimedia.org/r/186258 (owner: 10BBlack) [18:06:13] (03CR) 10Alexandros Kosiaris: [C: 04-1] "The technical part is obviously fine. That being said, /var/run a symlink to /run for all of precise, trusty and jessie. So to be honest, " [puppet] - 10https://gerrit.wikimedia.org/r/186258 (owner: 10BBlack) [18:09:12] RECOVERY - Disk space on antimony is OK: DISK OK [18:09:21] thanks to whoever did that :) [18:10:14] gzipped an apache log [18:13:03] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [18:17:23] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [18:18:02] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:18:33] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:18:33] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [18:18:42] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [18:20:42] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [18:28:17] 3Wikimedia-Logstash, operations, ops-core: Production hardware for Logstash service - https://phabricator.wikimedia.org/T84958#991282 (10bd808) >>! In T84958#988487, @Manybubbles wrote: >>>! In T84958#988453, @bd808 wrote: >> Here's my proposal: >> * Add 3 hosts matching the machine spec used for the MediaWiki E... [18:34:44] (03PS1) 10KartikMistry: WIP: Publish article to Main namespace for cawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186358 [18:36:53] <_joe_>  [18:39:02] 3operations: Re-image osmium and repurpose into VE testing host - https://phabricator.wikimedia.org/T87215#991317 (10ori) Procurement ticket: RT 9148 [18:53:55] (03CR) 10Ori.livneh: [C: 032] "OK'd by Giuseppe and Daniel; looks good to me as well." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186319 (owner: 10Jdlrobson) [18:55:10] ? [18:55:12] jenkins? [18:55:16] herro? [18:55:34] (03CR) 10Ori.livneh: [V: 032] Enable the wikidata mobile experiments on beta labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186319 (owner: 10Jdlrobson) [18:57:42] PROBLEM - ElasticSearch health check for shards on logstash1002 is CRITICAL: CRITICAL - elasticsearch inactive shards 15 threshold =0.1% breach: {ustatus: uyellow, unumber_of_nodes: 3, uunassigned_shards: 11, utimed_out: False, uactive_primary_shards: 41, ucluster_name: uproduction-logstash-eqiad, urelocating_shards: 0, uactive_shards: 108, uinitializing_shards: 4, unumber_of_data_nodes: 3} [18:57:52] PROBLEM - ElasticSearch health check for shards on logstash1001 is CRITICAL: CRITICAL - elasticsearch inactive shards 15 threshold =0.1% breach: {ustatus: uyellow, unumber_of_nodes: 3, uunassigned_shards: 11, utimed_out: False, uactive_primary_shards: 41, ucluster_name: uproduction-logstash-eqiad, urelocating_shards: 0, uactive_shards: 108, uinitializing_shards: 4, unumber_of_data_nodes: 3} [18:58:12] PROBLEM - ElasticSearch health check for shards on logstash1003 is CRITICAL: CRITICAL - elasticsearch inactive shards 15 threshold =0.1% breach: {ustatus: uyellow, unumber_of_nodes: 3, uunassigned_shards: 11, utimed_out: False, uactive_primary_shards: 41, ucluster_name: uproduction-logstash-eqiad, urelocating_shards: 0, uactive_shards: 108, uinitializing_shards: 4, unumber_of_data_nodes: 3} [18:58:46] (03PS4) 10BBlack: disable nginx udplog for SSL clusters [puppet] - 10https://gerrit.wikimedia.org/r/186257 [18:58:48] (03PS3) 10BBlack: use /run/ directly for nginx pidfile [puppet] - 10https://gerrit.wikimedia.org/r/186258 [18:59:44] (03CR) 10Giuseppe Lavagetto: [C: 032] snapshot: unify node declarations, use role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/185152 (owner: 10Giuseppe Lavagetto) [19:01:34] (03CR) 10Nikerabbit: [C: 04-1] WIP: Publish article to Main namespace for cawiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186358 (owner: 10KartikMistry) [19:02:24] !log ebernhardson Synchronized php-1.25wmf15/extensions/Flow/: Bump flow submodule in 1.25wmf15 (duration: 00m 08s) [19:02:38] Logged the message, Master [19:03:50] (03PS1) 10Krinkle: docroot/foundation: Remove unused and outdated 'FrameResize.html' utility [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186372 [19:03:51] !log ebernhardson Synchronized php-1.25wmf14/extensions/Flow: Bump flow submodule in 1.25wmf14 (duration: 00m 07s) [19:03:52] (03PS1) 10Krinkle: docroot/fundraising: Remove unused 'fundraising' landing page [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186373 [19:03:54] (03PS1) 10Krinkle: docroot/foundation: Remove unused 'levée_de_fonds.html' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186374 [19:03:56] (03PS3) 10Giuseppe Lavagetto: virt: use role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/185153 [19:04:01] Logged the message, Master [19:09:16] greg-g, hey [19:10:35] (03CR) 10BBlack: [C: 04-1] "This depends on (at least) If32afcc082ae28248eca46b58dc2748811b00489 ; currently this udplog traffic feeds http://stats.grok.se/" [puppet] - 10https://gerrit.wikimedia.org/r/186257 (owner: 10BBlack) [19:13:09] Krenair: hello [19:13:37] ah, looks like you commented on what I was going to ask about [19:13:42] never mind :) [19:13:50] :) [19:20:51] (03PS2) 10KartikMistry: WIP: Publish article to Main namespace for cawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186358 [19:25:18] 3operations: mysql boxes not in ganglia - https://phabricator.wikimedia.org/T87209#991468 (10Reedy) This makes https://noc.wikimedia.org/dbtree not actually show any replag stats etc [19:25:18] (03PS4) 10BBlack: use /run/ directly for nginx pidfile [puppet] - 10https://gerrit.wikimedia.org/r/186258 [19:26:31] (03PS2) 10Krinkle: docroot: Remove unused 'foundation/levée_de_fonds.html' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186374 [19:26:33] (03PS2) 10Krinkle: docroot: Remove unused 'foundation/fundraising' landing page [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186373 [19:26:35] (03PS2) 10Krinkle: docroot: Remove unused and outdated '/foundation/FrameResize.html' utility [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186372 [19:26:37] (03PS1) 10Krinkle: docroot: Provide HiDPI version of Logo in 'default' index [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186382 [19:26:57] lol [19:27:36] Krinkle: How empty is that docroot when you've finished with the above? [19:28:10] Reedy: not empty enoguh [19:28:16] boo :( [19:28:24] How did you confirm they weren't used anywhere? [19:28:26] Reedy: post-commit https://git.wikimedia.org/tree/operations%2Fmediawiki-config/2c59602dcf3031580a59193fc4e1f227fbdb1185/docroot%2Ffoundation [19:28:54] Reedy: wikimedia-git wide search, svn, google, and logic (seeing if it's actually accessible by url) [19:29:10] 'foundation/fundraising' is inaccessible due to HTTP redirect in puppet/apache-config [19:29:23] FrameResize.html is ancient and doesn't do anything, and isn't referenced from anywhere [19:29:49] (03CR) 10BBlack: [C: 032] use /run/ directly for nginx pidfile [puppet] - 10https://gerrit.wikimedia.org/r/186258 (owner: 10BBlack) [19:29:51] (03PS2) 10Krinkle: docroot: Provide HiDPI version of Logo in 'default' index [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186382 [19:30:10] (03CR) 10Tim Starling: "gmetad should not be running as nobody, that is a pretty serious bug. We have kernel.yama.ptrace_scope=1 (restricted) everywhere so that i" [puppet] - 10https://gerrit.wikimedia.org/r/174896 (owner: 10Hoo man) [19:30:24] Reedy: Do you know how to access mediawiki-config 404.html? [19:30:31] I can't find a way to trigger it [19:31:54] https://en.wikipedia.org/resources/sdfd https://en.wikipedia.org/w/resources/sdfd One says "Did you mean to type" the other "To check for". They look identical otherwise. Neither is 404.html, both look like w/404.php [19:32:07] I wonder when 404.html is used [19:32:19] secure.wm.o [19:32:27] https://github.com/wikimedia/operations-puppet/blob/2356ee6e3155a5a486452c9e37e6ff2a2f891e49/modules/mediawiki/files/apache/sites/secure.wikimedia.conf#L27 [19:32:31] Ha [19:32:40] https://secure.wikimedia.org/404.html [19:32:49] And by docroot default as well [19:32:52] That'simportant [19:33:06] Although http://krinkle.bits.beta.wmflabs.org/ - which is docroot/default, doesn't get it: http://krinkle.bits.beta.wmflabs.org/sdfsdf [19:33:11] eh? [19:33:12] https://github.com/wikimedia/operations-puppet/blob/2356ee6e3155a5a486452c9e37e6ff2a2f891e49/modules/mediawiki/files/apache/sites/nonexistent.conf#L5 [19:33:13] I guess because it's not using that aapche config [19:33:18] default is using index.html [19:33:25] Oh, right [19:33:29] well, taht's failing too then [19:33:41] I blame github search [19:33:42] https://github.com/wikimedia/operations-puppet/search?utf8=%E2%9C%93&q=404.html [19:33:45] it matched 404 index.html [19:33:48] for 404.html [19:33:54] * Krinkle adds quotes [19:33:55] heh, read the results :P [19:34:36] Reedy: What's with the differnet 404 php ones, any idea where those fragment? [19:34:42] symlinks? [19:34:45] E.g. https://en.wikipedia.org/resources/sdfd https://en.wikipedia.org/w/resources/sdfd [19:35:01] Neither is 404.html, but they can't both be 404.php because they are different in their text slightly [19:36:02] it's a conditional in 404.php [19:36:03] https://github.com/wikimedia/operations-mediawiki-config/blob/master/w/404.php#L15-L30 [19:36:11] (03PS1) 10Krinkle: docroot: Update default 404.html to match new default index.html style [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186384 [19:36:38] Reedy: Oh.. [19:36:41] I hadn't considered that [19:37:17] akosiaris: around? [19:37:27] He's in a meeting kart_, he might respond [19:37:29] Krinkle: I think we should optimise those regexes [19:37:48] doesn't really need to be 6 seperate ones [19:38:09] it could be 3 conservatively [19:38:18] * Reedy pokes [19:38:46] Reedy: thanks. Having Mars from the planet UK :) [19:40:05] 3ops-codfw, operations: racktables output for codfw - https://phabricator.wikimedia.org/T86019#991522 (10RobH) can this be resolved? [19:41:15] 3ops-codfw, operations: racktables output for codfw - https://phabricator.wikimedia.org/T86019#991530 (10Cmjohnson) 5Open>3Resolved Completed [19:41:49] greg-g, we've fixed https://phabricator.wikimedia.org/T87396 so it can't happen again. However, we'd like to clean up any pages with content or deleted content in the wrong namespace. [19:42:54] To do so, we want to put some temporary code in production on mw1017. We'll put the patch up on Gerrit once we figure it out. [19:49:25] Hey guys, mobile is doing a no-op deployment currently for a deployment training (already have greg’s approval). Just a head’s up :) [19:50:58] (03PS1) 10Reedy: Condense 404 handling regexes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186392 [19:51:03] Krinkle: ^ [19:51:12] kart_: in meetings for most of the day. I 'll contact you if I get some time free [19:52:07] (03CR) 10Krinkle: [C: 031] Condense 404 handling regexes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186392 (owner: 10Reedy) [19:54:09] (03CR) 10Reedy: [C: 031] conf: Clean up highlight.php and fix http-redirect bug [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186327 (owner: 10Krinkle) [19:55:22] 3operations: Remove misc/statistics.pp - https://phabricator.wikimedia.org/T87450#991561 (10Ottomata) 3NEW a:3Ottomata [19:55:48] (03CR) 10Reedy: "Can we just kill wmf_logo_with_text.png too?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186382 (owner: 10Krinkle) [19:57:25] (03PS3) 10KartikMistry: Use cxserver/deploy in deployment [puppet] - 10https://gerrit.wikimedia.org/r/184217 [19:57:59] (03PS3) 10Krinkle: docroot: Provide HiDPI version of Logo in 'default' index [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186382 [20:02:35] 3operations: Remove misc/statistics.pp - https://phabricator.wikimedia.org/T87450#991617 (10Ottomata) [20:04:07] (03CR) 10Krinkle: "Done" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186382 (owner: 10Krinkle) [20:04:19] hm paravoid says logstash is freaking out. *looks* [20:06:56] jgage: hmm? [20:06:59] bd808: ^^ [20:07:28] 3operations: add direct to task emailing for onsite queues (only for vendors!) - https://phabricator.wikimedia.org/T87454#991638 (10RobH) 3NEW a:3RobH [20:08:45] reedy, there are 4 shards initializing. perhaps it's already in recovery? i see a large volume of network traffic from logstash1001 to 02, suggesting resync [20:09:10] !log deployed patch for T64685 [20:09:23] Logged the message, Master [20:10:11] bd808|BUFFER: Looks like https://logstash.wikimedia.org/#/dashboard/elasticsearch/fatalmonitor is broken [20:10:32] PROBLEM - puppet last run on labnet1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:10:59] we should get logstash100* upgraded to 14.04 so we can use the wrapper scripts ^d wrote [20:11:15] <^d> Fan-f'ing tastic. [20:11:49] kaldari: yeah, jgage and reedy are looking into it [20:15:59] (03PS1) 10Ottomata: First commit in a multi-commit effort to move misc/statistics.pp into modules/ [puppet] - 10https://gerrit.wikimedia.org/r/186396 [20:16:22] * jgage is chatting with bd808 and ^d about logstash [20:16:57] (03CR) 10jenkins-bot: [V: 04-1] First commit in a multi-commit effort to move misc/statistics.pp into modules/ [puppet] - 10https://gerrit.wikimedia.org/r/186396 (owner: 10Ottomata) [20:17:14] yay in person :) [20:17:19] :D [20:18:38] short answer is that logstash is recovering, we're discussing longer term solution [20:20:35] (03PS2) 10Ottomata: First commit in a multi-commit effort to move misc/statistics.pp into modules/ [puppet] - 10https://gerrit.wikimedia.org/r/186396 [20:27:12] RECOVERY - puppet last run on labnet1001 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [20:30:07] !log rmoen Synchronized php-1.25wmf15/extensions/MobileFrontend/: updating mobilefrontend credits (duration: 00m 06s) [20:30:16] Logged the message, Master [20:35:19] ^^ that's kaldari/max training sam smith [20:36:03] PROBLEM - puppet last run on mw1001 is CRITICAL: CRITICAL: Puppet has 1 failures [20:38:40] greg-g: also training robmoen for good measure :) [20:38:45] :) :) [20:54:02] RECOVERY - puppet last run on mw1001 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [20:58:13] (03CR) 10Cscott: "Sigh. Parsoid roots now have root on the parsoid boxes (I9b9919eb45f184d34b30d4a8d3dc113e00c2633d) but that still doesn't give parsoid *a" [puppet] - 10https://gerrit.wikimedia.org/r/180221 (https://phabricator.wikimedia.org/T86804) (owner: 10Cscott) [20:59:30] (03CR) 10Ottomata: [C: 032] First commit in a multi-commit effort to move misc/statistics.pp into modules/ [puppet] - 10https://gerrit.wikimedia.org/r/186396 (owner: 10Ottomata) [21:00:19] (03PS3) 10Cscott: Give parsoid admins the ability to update/restart the RT testing service. [puppet] - 10https://gerrit.wikimedia.org/r/180221 (https://phabricator.wikimedia.org/T86804) [21:01:05] (03PS1) 10Ottomata: Fix type in class name include [puppet] - 10https://gerrit.wikimedia.org/r/186411 [21:01:07] (03PS1) 10RobH: add direct to task emailing for ops onsite dc projects [puppet] - 10https://gerrit.wikimedia.org/r/186412 [21:01:19] (03PS2) 10Ottomata: Fix typo in class name include [puppet] - 10https://gerrit.wikimedia.org/r/186411 [21:01:20] 3Ops-Access-Requests, operations: Give parsoid admins the ability to update/restart the RT testing service - https://phabricator.wikimedia.org/T86804#991750 (10cscott) Still hasn't happened yet. Gabriel added a patch to give *parsoid-roots* root, but that doesn't help parsoid *admin*s at all. So we still need... [21:01:37] (03PS2) 10RobH: cleaning out holmium entries [puppet] - 10https://gerrit.wikimedia.org/r/186357 [21:01:40] (03CR) 10Ottomata: [C: 032 V: 032] Fix typo in class name include [puppet] - 10https://gerrit.wikimedia.org/r/186411 (owner: 10Ottomata) [21:02:08] (03CR) 10jenkins-bot: [V: 04-1] add direct to task emailing for ops onsite dc projects [puppet] - 10https://gerrit.wikimedia.org/r/186412 (owner: 10RobH) [21:02:52] (03CR) 10RobH: [C: 032] cleaning out holmium entries [puppet] - 10https://gerrit.wikimedia.org/r/186357 (owner: 10RobH) [21:02:55] (03PS1) 10Ottomata: Globally qualify statistics module include [puppet] - 10https://gerrit.wikimedia.org/r/186413 [21:03:02] (03PS2) 10Ottomata: Globally qualify statistics module include [puppet] - 10https://gerrit.wikimedia.org/r/186413 [21:03:13] (03PS3) 10RobH: cleaning out holmium entries [puppet] - 10https://gerrit.wikimedia.org/r/186357 [21:03:13] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: puppet fail [21:03:17] (03CR) 10Ottomata: [C: 032 V: 032] Globally qualify statistics module include [puppet] - 10https://gerrit.wikimedia.org/r/186413 (owner: 10Ottomata) [21:03:49] (03CR) 10RobH: [C: 032] cleaning out holmium entries [puppet] - 10https://gerrit.wikimedia.org/r/186357 (owner: 10RobH) [21:04:20] (03PS2) 10RobH: add direct to task emailing for ops onsite dc projects [puppet] - 10https://gerrit.wikimedia.org/r/186412 [21:05:04] (03CR) 10jenkins-bot: [V: 04-1] add direct to task emailing for ops onsite dc projects [puppet] - 10https://gerrit.wikimedia.org/r/186412 (owner: 10RobH) [21:07:01] (03PS3) 10RobH: add direct to task emailing for ops onsite dc projects [puppet] - 10https://gerrit.wikimedia.org/r/186412 [21:07:18] (03PS1) 10Ottomata: Make misc::statistics::user and misc::statistics::base include module classes [puppet] - 10https://gerrit.wikimedia.org/r/186417 [21:07:39] (03PS2) 10Ottomata: Make misc::statistics::user and misc::statistics::base include module classes [puppet] - 10https://gerrit.wikimedia.org/r/186417 [21:07:45] (03CR) 10jenkins-bot: [V: 04-1] add direct to task emailing for ops onsite dc projects [puppet] - 10https://gerrit.wikimedia.org/r/186412 (owner: 10RobH) [21:07:58] (03PS4) 10RobH: add direct to task emailing for ops onsite dc projects [puppet] - 10https://gerrit.wikimedia.org/r/186412 [21:08:53] (03CR) 10jenkins-bot: [V: 04-1] add direct to task emailing for ops onsite dc projects [puppet] - 10https://gerrit.wikimedia.org/r/186412 (owner: 10RobH) [21:09:41] (03CR) 10Ottomata: [C: 032] Make misc::statistics::user and misc::statistics::base include module classes [puppet] - 10https://gerrit.wikimedia.org/r/186417 (owner: 10Ottomata) [21:10:00] robh, merging. [21:10:25] (03PS1) 10Filippo Giunchedi: fix check_elasticsearch CRITICAL output [puppet] - 10https://gerrit.wikimedia.org/r/186418 [21:10:34] (03Abandoned) 10RobH: add direct to task emailing for ops onsite dc projects [puppet] - 10https://gerrit.wikimedia.org/r/186412 (owner: 10RobH) [21:10:56] (03Abandoned) 10RobH: subscribe the chained.pem file to the non-chained.pem file [puppet] - 10https://gerrit.wikimedia.org/r/131087 (owner: 10RobH) [21:12:14] (03PS1) 10Ottomata: Just make misc::statistics::base depend on ::statistics [puppet] - 10https://gerrit.wikimedia.org/r/186419 [21:13:57] (03CR) 10Ottomata: [C: 032] Just make misc::statistics::base depend on ::statistics [puppet] - 10https://gerrit.wikimedia.org/r/186419 (owner: 10Ottomata) [21:14:35] (03PS1) 10Phuedx: Replace phuedx's key [puppet] - 10https://gerrit.wikimedia.org/r/186420 [21:15:15] who should i offer up my soul to get that merged ^ [21:15:20] i am in the office [21:15:27] i can offer chocolate stolen from the kitchen [21:15:32] freshly brewed coffee [21:15:35] beans are from berlin [21:15:36] oh right, chocolate! [21:16:08] i lost the password for my old key [21:16:56] (03PS1) 10Ottomata: Temporarily se local variables in misc statistics classes to satisfy existing class references [puppet] - 10https://gerrit.wikimedia.org/r/186421 [21:17:12] (03PS2) 10Ottomata: Temporarily use local variables in misc statistics classes to satisfy existing class references [puppet] - 10https://gerrit.wikimedia.org/r/186421 [21:17:54] (03CR) 10Ottomata: [C: 032 V: 032] Temporarily use local variables in misc statistics classes to satisfy existing class references [puppet] - 10https://gerrit.wikimedia.org/r/186421 (owner: 10Ottomata) [21:21:01] <_joe_> !log uploaded new hhvm package to apt.w.o [21:21:14] Logged the message, Master [21:22:17] (03PS1) 10Ottomata: Duplicate variable definition during migration to stat module [puppet] - 10https://gerrit.wikimedia.org/r/186422 [21:23:09] (03CR) 10Ottomata: [C: 032] Duplicate variable definition during migration to stat module [puppet] - 10https://gerrit.wikimedia.org/r/186422 (owner: 10Ottomata) [21:24:52] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [21:25:34] is there a magic way to make phab not auto-link a string like "R420"? In my case it's a Dell model number but Phab is auto-linking it to https://phabricator.wikimedia.org/diffusion/EPBO/ for reasons unclear to me. [21:25:40] phuedx: we'll need you to verify your identity somehow. One option is for you to edit your user page in a staff account. Or if there's someone in the Ops team who would recognize you you can do a quick video call. [21:25:49] ^ chasemp [21:26:22] PROBLEM - puppet last run on helium is CRITICAL: CRITICAL: puppet fail [21:26:48] (03PS1) 10Giuseppe Lavagetto: mediawiki: define pcre_cache_type for canary appservers [puppet] - 10https://gerrit.wikimedia.org/r/186453 [21:27:07] jgage: `literal string` [21:27:08] or [21:27:15] <_joe_> ori, TimStarling: I'm going to upload the new package to the first subset of appservers [21:27:16] %%%literal string%%% [21:27:26] <_joe_> it has the new pcre cache code [21:27:29] https://secure.phabricator.com/book/phabricator/article/remarkup/ [21:27:33] phuedx: are you in the office now? [21:27:58] ah yeah thanks chasemp [21:28:11] totally trivial but nice to know [21:28:16] _joe_: good stuff [21:28:26] phuedx: If so, just stick your head in Pliny and look for me :) [21:29:05] <_joe_> TimStarling: hopefully so. It's been on beta for a week, it should be stable enough [21:29:47] that backported patch that I prepared? [21:30:04] 3operations: Allocate temporary Elasticsearch nodes from spares pool for Logstash - https://phabricator.wikimedia.org/T87460#991788 (10Gage) 3NEW a:3RobH [21:30:21] (03PS2) 10Giuseppe Lavagetto: mediawiki: define pcre_cache_type for canary appservers [puppet] - 10https://gerrit.wikimedia.org/r/186453 [21:31:02] <_joe_> TimStarling: that, and a fix for the leak that made account unification choke on OOM [21:31:14] right [21:31:18] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki: define pcre_cache_type for canary appservers [puppet] - 10https://gerrit.wikimedia.org/r/186453 (owner: 10Giuseppe Lavagetto) [21:31:48] <_joe_> TimStarling: what's the default cache type, if you remember that? [21:32:08] <_joe_> or I'll look at the code [21:32:09] in my backport, it should be scalable [21:32:14] <_joe_> ok [21:32:19] in the one that was merged to upstream, it is static I think [21:32:21] <_joe_> we should use "lru", right? [21:32:28] yes [21:32:31] <_joe_> ok [21:32:50] <_joe_> scalable is still not disastrous, so I can install the package and run puppet afterwards [21:33:01] yes [21:33:49] Brett said the difference between scalable and lru was too small to measure in his testing [21:34:00] <_joe_> good to know [21:34:28] what is scalable, again? [21:34:34] <_joe_> !log installing a new hhvm package on the canary appservers [21:34:38] Logged the message, Master [21:35:54] 3Ops-Access-Requests: Cluster deploy access for Andrew Russell Green - https://phabricator.wikimedia.org/T87463#991821 (10Dzahn) 3NEW [21:36:26] 3Ops-Access-Requests: Cluster deploy access for Andrew Russell Green - https://phabricator.wikimedia.org/T87463#991828 (10Dzahn) a:3RobH [21:37:08] 3Ops-Access-Requests: Cluster deploy access for Andrew Russell Green - https://phabricator.wikimedia.org/T87463#991821 (10Dzahn) [21:37:42] 3Scrum-of-Scrums, RESTBase, operations, Services: Restbase deployment - https://phabricator.wikimedia.org/T1228#991836 (10GWicke) [21:38:00] 3Scrum-of-Scrums, RESTBase, operations, Services: Restbase deployment - https://phabricator.wikimedia.org/T1228#21142 (10GWicke) [21:38:12] mutante: i wasnt going ot pull that one in [21:38:18] since no one merged my weeksold change [21:38:18] heh [21:38:29] but ill comment on phab, im just merging that shit though cuz i just required him to do all the old stuff [21:38:35] i was about to review it :) [21:38:40] awesome! [21:38:42] thank you please do [21:38:44] i wanted to check RT one last time [21:38:48] https://gerrit.wikimedia.org/r/#/c/183403/2 [21:38:49] and saw exactly 1 ticket left [21:38:50] heh [21:38:56] (03PS1) 10Ottomata: Apply new statistics::compute class to stat1003 [puppet] - 10https://gerrit.wikimedia.org/r/186496 [21:39:20] (03PS1) 10BBlack: Explicit provider for ganglia_new::monitor::service [puppet] - 10https://gerrit.wikimedia.org/r/186497 [21:40:33] (03CR) 10Ottomata: [C: 032] Apply new statistics::compute class to stat1003 [puppet] - 10https://gerrit.wikimedia.org/r/186496 (owner: 10Ottomata) [21:40:49] _joe_: merging. [21:40:52] PROBLEM - puppet last run on mw1019 is CRITICAL: CRITICAL: Puppet has 1 failures [21:41:16] mediawiki: define pcre_cache_type for canary appservers (ccba8c4) [21:42:18] quick question - is there a page on wikitech.wikimedia that lists what version of MediaWiki each WMF project is running ? [21:42:40] ahh shit i had an unmerged on palladium blog change [21:42:43] it wont break anything [21:42:49] just sorry to whoever had to read it ;D [21:42:53] ottomata: I imagine its you [21:43:07] i merged it :) [21:43:14] thx =] [21:44:07] (03CR) 10Dzahn: [C: 031] "UID matches labs user, key looks good like on https://office.wikimedia.org/w/index.php?title=User:AGreen_%28WMF%29&oldid=126288 , ticket c" [puppet] - 10https://gerrit.wikimedia.org/r/183403 (owner: 10RobH) [21:44:11] robh: also , test ticket for the process [21:44:30] well, i specifially wasnt going to make andrew green do all that shit since it was waiting for merge and approved in rt [21:44:32] seems mean [21:44:43] i just forgot he was there [21:44:44] that's true, it's already approved on RT [21:44:50] (03PS2) 10BBlack: Explicit provider for ganglia_new::monitor::service [puppet] - 10https://gerrit.wikimedia.org/r/186497 [21:44:52] yea, lets just squeak that one by in rt [21:44:56] or 'pre new rules' [21:45:03] whatevs, and im willing ot take heat, heh [21:45:05] (03CR) 10BBlack: [C: 032 V: 032] Explicit provider for ganglia_new::monitor::service [puppet] - 10https://gerrit.wikimedia.org/r/186497 (owner: 10BBlack) [21:45:06] and he also did the ACK i read the document thing [21:45:08] on RT [21:45:13] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: puppet fail [21:45:15] yes [21:45:38] hrmm [21:45:51] (03CR) 10RobH: [C: 032] granting cluster deployment access to Andrew Green [puppet] - 10https://gerrit.wikimedia.org/r/183403 (owner: 10RobH) [21:45:58] 2. Read, comprehend, and sign the [https://phabricator.wikimedia.org/L3 Acknowledgement of Wikimedia Server Access Responsibilities] document [21:46:01] bah,wrong paste [21:46:06] (03PS1) 10Ottomata: Comment out manual include of misc::statistics::packages::python in geowiki classes [puppet] - 10https://gerrit.wikimedia.org/r/186499 [21:46:12] NotASpy, don't know about wikitech, but you can work it out from https://www.mediawiki.org/wiki/MediaWiki_1.25/Roadmap [21:46:13] The change could not be rebased due to a path conflict during merge [21:46:17] for my agreen patch [21:46:19] grrr [21:46:41] (03PS2) 10Ottomata: Comment out manual include of misc::statistics::packages::python in geowiki classes [puppet] - 10https://gerrit.wikimedia.org/r/186499 [21:46:46] (03PS3) 10Ottomata: Comment out manual include of misc::statistics::packages::python in geowiki classes [puppet] - 10https://gerrit.wikimedia.org/r/186499 [21:46:55] (03CR) 10Ottomata: [C: 032 V: 032] Comment out manual include of misc::statistics::packages::python in geowiki classes [puppet] - 10https://gerrit.wikimedia.org/r/186499 (owner: 10Ottomata) [21:47:02] Krenair: that looks perfect. Trying to help someone work out why Twinkle won't work on another (WMF) project [21:47:14] spanish wikipedia? [21:47:23] (03PS3) 10Dzahn: granting cluster deployment access to Andrew Green [puppet] - 10https://gerrit.wikimedia.org/r/183403 (https://phabricator.wikimedia.org/T87463) (owner: 10RobH) [21:47:30] (03CR) 10jenkins-bot: [V: 04-1] granting cluster deployment access to Andrew Green [puppet] - 10https://gerrit.wikimedia.org/r/183403 (https://phabricator.wikimedia.org/T87463) (owner: 10RobH) [21:47:49] mutante: This change was unable to be automatically merged with the current state of the repository. Please rebase your change and upload a new patchset. [21:48:05] im about to just blow it out and redo via copy and paste =P [21:48:16] hold on [21:48:23] i just edited the message but lemme try fixing it [21:48:29] Krenair: yeah [21:48:29] cool [21:48:51] I don't know if version of mediawiki would cause it, NotASpy [21:48:53] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: puppet fail [21:49:22] Krenair: I didn't think it would either, I think it'll be a language thing somewhere [21:49:49] NotASpy, did you check the JS console? [21:49:57] <_joe_> ottomata: that wasn't really cool to merge at that moment, but well, it's done [21:50:07] (03PS1) 10Ottomata: Comment out uses of researchdb password classes [puppet] - 10https://gerrit.wikimedia.org/r/186500 [21:50:31] oop, _joe_ sorry about that, should it have been merged in gerrit then? [21:50:47] (03PS2) 10Ottomata: Comment out uses of researchdb password classes [puppet] - 10https://gerrit.wikimedia.org/r/186500 [21:50:55] Krenair: I've not looked. I can't do much as I don't have an account on Spanish Wikipedia (SUL issues and my lazyness) [21:51:11] mutante: so lemme know if you cannot or give up cuz i have a new fresh patch for it ;D [21:51:53] NotASpy, that really needs to be the first place you check when looking into why a user gadget is broken [21:52:03] (03PS4) 10Dzahn: granting cluster deployment access to Andrew Green [puppet] - 10https://gerrit.wikimedia.org/r/183403 (https://phabricator.wikimedia.org/T87463) (owner: 10RobH) [21:52:27] (03CR) 10Ottomata: [C: 032] Comment out uses of researchdb password classes [puppet] - 10https://gerrit.wikimedia.org/r/186500 (owner: 10Ottomata) [21:52:31] yay [21:52:31] robh: there [21:52:41] (03CR) 10RobH: [C: 031] granting cluster deployment access to Andrew Green [puppet] - 10https://gerrit.wikimedia.org/r/183403 (https://phabricator.wikimedia.org/T87463) (owner: 10RobH) [21:53:02] (03CR) 10RobH: [C: 032] granting cluster deployment access to Andrew Green [puppet] - 10https://gerrit.wikimedia.org/r/183403 (https://phabricator.wikimedia.org/T87463) (owner: 10RobH) [21:53:07] mutante: thx dude [21:53:10] merged~! [21:53:27] i was about to ask who does, yw [21:53:45] so that was it with RT tickets for access [21:54:03] PROBLEM - puppet last run on mw1116 is CRITICAL: CRITICAL: Puppet has 1 failures [21:54:28] 3Ops-Access-Requests: Cluster deploy access for Andrew Russell Green - https://phabricator.wikimedia.org/T87463#991875 (10RobH) 5Open>3Resolved So this was actually handled in RT9060 as well, and is now live via https://gerrit.wikimedia.org/r/#/c/183403/ resolving! [21:54:52] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [21:55:22] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: puppet fail [21:55:24] Ok, I've updated the https://wikitech.wikimedia.org/wiki/Requesting_shell_access and https://phabricator.wikimedia.org/project/view/956/ (ops-access-requests) [21:55:50] !log removed puppet stored config for search* and searchidx* [21:55:58] (03PS1) 10Ottomata: Comment out includes that are now covered by the new role [puppet] - 10https://gerrit.wikimedia.org/r/186501 [21:56:01] Logged the message, Master [21:56:03] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [21:56:42] (03CR) 10Ottomata: [C: 032 V: 032] Comment out includes that are now covered by the new role [puppet] - 10https://gerrit.wikimedia.org/r/186501 (owner: 10Ottomata) [21:58:35] chasemp: the admin doesnt have the create project ability!?? [21:58:45] robh: do we want a project tag for "domains"? [21:59:33] robh: better to create projects as you since it's auditable I suppose [21:59:45] Did we just deploy a Translate change? [21:59:48] heh [22:00:04] https://www.mediawiki.org/wiki/API:Options [22:00:11] http://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects [22:00:12] 3ops-core: Build a new HHVM package - https://phabricator.wikimedia.org/T86906#991896 (10Joe) Package is on apt.w.o and deployed to the canary appservers. It will go online everywhere if it's been stable over the weekend. [22:00:30] (03PS1) 10Ottomata: Move custom researcher group researchdb password file definition into cruncher role class [puppet] - 10https://gerrit.wikimedia.org/r/186502 [22:00:36] RECOVERY - ElasticSearch health check for shards on logstash1002 is OK: OK - elasticsearch status production-logstash-eqiad: status: yellow, number_of_nodes: 3, unassigned_shards: 8, timed_out: False, active_primary_shards: 41, cluster_name: production-logstash-eqiad, relocating_shards: 0, active_shards: 111, initializing_shards: 4, number_of_data_nodes: 3 [22:00:44] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: Puppet has 1 failures [22:00:54] RECOVERY - ElasticSearch health check for shards on logstash1001 is OK: OK - elasticsearch status production-logstash-eqiad: status: yellow, number_of_nodes: 3, unassigned_shards: 8, timed_out: False, active_primary_shards: 41, cluster_name: production-logstash-eqiad, relocating_shards: 0, active_shards: 111, initializing_shards: 4, number_of_data_nodes: 3 [22:00:55] PROBLEM - puppet last run on mw1020 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:04] PROBLEM - puppet last run on mw1017 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:07] akosiaris: Hey, could we get https://gerrit.wikimedia.org/r/#/c/184904/ merged do you think? [22:01:16] robh: " create a new task under Project-Creators " :p [22:01:24] RECOVERY - ElasticSearch health check for shards on logstash1003 is OK: OK - elasticsearch status production-logstash-eqiad: status: yellow, number_of_nodes: 3, unassigned_shards: 8, timed_out: False, active_primary_shards: 41, cluster_name: production-logstash-eqiad, relocating_shards: 0, active_shards: 111, initializing_shards: 4, number_of_data_nodes: 3 [22:01:43] yes best practice (non urgent need) is to make a ticket for a project [22:01:45] to make sure it's not a dupe [22:01:50] or covered somewhere else [22:01:50] (03PS2) 10Ottomata: Move custom researcher group researchdb password file definition into node definition [puppet] - 10https://gerrit.wikimedia.org/r/186502 [22:01:54] I don't even know all of them at this point [22:02:17] some are "ops" , some are "Ops" [22:02:41] (03CR) 10Ottomata: [C: 032] Move custom researcher group researchdb password file definition into node definition [puppet] - 10https://gerrit.wikimedia.org/r/186502 (owner: 10Ottomata) [22:03:04] what makes sense for the domain stuff, hmmmm [22:03:13] also under operations and legal [22:04:45] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [22:05:15] PROBLEM - puppet last run on mw1115 is CRITICAL: CRITICAL: Puppet has 1 failures [22:05:42] mutante: yea im making it as me [22:05:44] (03PS1) 10Ottomata: Include more cruncher classes in new role classes [puppet] - 10https://gerrit.wikimedia.org/r/186504 [22:07:44] (03CR) 10Ottomata: [C: 032] Include more cruncher classes in new role classes [puppet] - 10https://gerrit.wikimedia.org/r/186504 (owner: 10Ottomata) [22:08:15] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: Puppet has 1 failures [22:09:34] (03PS1) 10Ottomata: Fix dependency for class statistics::rsync::eventlogging [puppet] - 10https://gerrit.wikimedia.org/r/186506 [22:09:50] (03CR) 10Ottomata: [C: 032 V: 032] Fix dependency for class statistics::rsync::eventlogging [puppet] - 10https://gerrit.wikimedia.org/r/186506 (owner: 10Ottomata) [22:11:45] RECOVERY - puppet last run on mw1116 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [22:15:00] (03PS1) 10Ottomata: role::statistics::cruncher now uses module classes (except for geowiki) [puppet] - 10https://gerrit.wikimedia.org/r/186508 [22:16:14] (03CR) 10Ottomata: [C: 032] role::statistics::cruncher now uses module classes (except for geowiki) [puppet] - 10https://gerrit.wikimedia.org/r/186508 (owner: 10Ottomata) [22:18:04] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [22:18:25] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [22:18:34] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [22:18:44] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [22:19:35] (03PS1) 10Ottomata: role::statistics::private now uses module includes [puppet] - 10https://gerrit.wikimedia.org/r/186509 [22:20:49] (03CR) 10Ottomata: [C: 032] role::statistics::private now uses module includes [puppet] - 10https://gerrit.wikimedia.org/r/186509 (owner: 10Ottomata) [22:22:15] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [22:24:35] RECOVERY - Disk space on labstore1001 is OK: DISK OK [22:25:19] (03PS1) 10Ottomata: Use statistics::web class [puppet] - 10https://gerrit.wikimedia.org/r/186510 [22:25:46] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [22:26:22] (03CR) 10Nikerabbit: "Still WIP?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186358 (owner: 10KartikMistry) [22:27:38] Krinkle: Shall we merge all the things? [22:27:44] Sure... [22:28:32] (03CR) 10Ottomata: [C: 032] Use statistics::web class [puppet] - 10https://gerrit.wikimedia.org/r/186510 (owner: 10Ottomata) [22:29:36] (03PS1) 10Filippo Giunchedi: lsearchd: reclaim search hosts [puppet] - 10https://gerrit.wikimedia.org/r/186511 (https://phabricator.wikimedia.org/T86149) [22:29:57] (03CR) 10Reedy: [C: 032] docroot: Remove unused and outdated '/foundation/FrameResize.html' utility [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186372 (owner: 10Krinkle) [22:30:02] godog: hey, got a minute? [22:30:05] (03Merged) 10jenkins-bot: docroot: Remove unused and outdated '/foundation/FrameResize.html' utility [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186372 (owner: 10Krinkle) [22:30:21] ori: sure [22:30:27] (03CR) 10Reedy: [C: 032] docroot: Remove unused 'foundation/fundraising' landing page [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186373 (owner: 10Krinkle) [22:30:32] (03Merged) 10jenkins-bot: docroot: Remove unused 'foundation/fundraising' landing page [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186373 (owner: 10Krinkle) [22:30:49] (03CR) 10Reedy: [C: 032] docroot: Remove unused 'foundation/levée_de_fonds.html' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186374 (owner: 10Krinkle) [22:30:54] (03Merged) 10jenkins-bot: docroot: Remove unused 'foundation/levée_de_fonds.html' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186374 (owner: 10Krinkle) [22:31:04] (03CR) 10Krinkle: [C: 032] Condense 404 handling regexes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186392 (owner: 10Reedy) [22:31:10] (03Merged) 10jenkins-bot: Condense 404 handling regexes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186392 (owner: 10Reedy) [22:31:18] (03PS4) 10Reedy: docroot: Provide HiDPI version of Logo in 'default' index [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186382 (owner: 10Krinkle) [22:31:24] (03CR) 10Reedy: [C: 032] docroot: Provide HiDPI version of Logo in 'default' index [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186382 (owner: 10Krinkle) [22:31:28] (03Merged) 10jenkins-bot: docroot: Provide HiDPI version of Logo in 'default' index [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186382 (owner: 10Krinkle) [22:31:34] 3operations: have any task put into ops-access-requests automatically generate an ops-access-review task - https://phabricator.wikimedia.org/T87467#991959 (10RobH) 3NEW a:3mmodell [22:31:35] PROBLEM - puppet last run on stat1001 is CRITICAL: CRITICAL: puppet fail [22:32:03] 3Phabricator, operations: have any task put into ops-access-requests automatically generate an ops-access-review task - https://phabricator.wikimedia.org/T87467#991967 (10chasemp) [22:32:36] (03PS2) 10Reedy: docroot: Update default 404.html to match new default index.html style [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186384 (owner: 10Krinkle) [22:32:41] (03PS1) 10Ottomata: role::statistics::web now uses module classes [puppet] - 10https://gerrit.wikimedia.org/r/186512 [22:32:43] (03CR) 10Reedy: [C: 032] docroot: Update default 404.html to match new default index.html style [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186384 (owner: 10Krinkle) [22:32:51] (03Merged) 10jenkins-bot: docroot: Update default 404.html to match new default index.html style [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186384 (owner: 10Krinkle) [22:33:11] (03PS3) 10Reedy: conf: Clean up highlight.php and fix http-redirect bug [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186327 (owner: 10Krinkle) [22:33:17] (03CR) 10Reedy: [C: 032] conf: Clean up highlight.php and fix http-redirect bug [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186327 (owner: 10Krinkle) [22:33:22] (03Merged) 10jenkins-bot: conf: Clean up highlight.php and fix http-redirect bug [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186327 (owner: 10Krinkle) [22:33:54] wow, merge spam [22:33:56] (03PS2) 10Ottomata: role::statistics::web now uses module classes [puppet] - 10https://gerrit.wikimedia.org/r/186512 [22:34:03] (03CR) 10Ottomata: [C: 032 V: 032] role::statistics::web now uses module classes [puppet] - 10https://gerrit.wikimedia.org/r/186512 (owner: 10Ottomata) [22:34:16] (03PS3) 10Alexandros Kosiaris: Re-use Parsoid Varnishes for citoid too [puppet] - 10https://gerrit.wikimedia.org/r/184904 (https://phabricator.wikimedia.org/T76949) (owner: 10Jforrester) [22:34:56] (03CR) 10Reedy: "Have we got a deployment schedule for this? :)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186242 (owner: 1001tonythomas) [22:35:59] (03CR) 10Nemo bis: "greg-g, did you receive the request for this?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186242 (owner: 1001tonythomas) [22:37:46] (03PS1) 10Ottomata: Use proper path for statistics::sites templates [puppet] - 10https://gerrit.wikimedia.org/r/186513 [22:37:53] (03PS2) 10Ottomata: Use proper path for statistics::sites templates [puppet] - 10https://gerrit.wikimedia.org/r/186513 [22:38:11] !log reedy Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 05s) [22:38:16] (03PS3) 10Jforrester: Beta Features: Disable the Compact Personal Bar feature [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185116 (https://phabricator.wikimedia.org/T85541) [22:38:17] Logged the message, Master [22:38:26] (03CR) 10Jforrester: [C: 031] "Now good to go whenever." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185116 (https://phabricator.wikimedia.org/T85541) (owner: 10Jforrester) [22:39:15] (03PS1) 10Reedy: Update interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186514 [22:39:26] Nemo_bis: ^^ [22:39:31] (03CR) 10Reedy: [C: 032] Update interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186514 (owner: 10Reedy) [22:39:41] (03CR) 10Nemo bis: [C: 031] Beta Features: Disable the Compact Personal Bar feature [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185116 (https://phabricator.wikimedia.org/T85541) (owner: 10Jforrester) [22:39:45] (03Merged) 10jenkins-bot: Update interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186514 (owner: 10Reedy) [22:40:06] (03PS3) 10Ottomata: Use proper path for statistics::sites templates [puppet] - 10https://gerrit.wikimedia.org/r/186513 [22:40:14] ffs, I can't edit https://meta.wikimedia.org/wiki/Interwiki_map [22:40:17] (03PS1) 10Dzahn: puppetize SSH private key for jenkins VE sync [puppet] - 10https://gerrit.wikimedia.org/r/186515 [22:40:39] (03PS2) 10Dzahn: puppetize SSH private key for jenkins VE sync [puppet] - 10https://gerrit.wikimedia.org/r/186515 (https://phabricator.wikimedia.org/T84731) [22:40:56] !log reedy Synchronized docroot and w: Update docroot files (duration: 00m 05s) [22:40:58] Reedy: if you !log, they see it [22:41:05] And thanks :) [22:41:07] Logged the message, Master [22:41:08] [22:38:11] !log reedy Synchronized wmf-config/interwiki.cdb: Updating interwiki cache (duration: 00m 05s) [22:41:19] Yep, so you're fine :) [22:41:38] !log reedy Synchronized 404.html: updates (duration: 00m 06s) [22:41:40] (03CR) 10Ottomata: [C: 032] Use proper path for statistics::sites templates [puppet] - 10https://gerrit.wikimedia.org/r/186513 (owner: 10Ottomata) [22:41:47] Logged the message, Master [22:43:24] (03Abandoned) 10Reedy: $wgContentHandlerUseDB true everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170129 (https://phabricator.wikimedia.org/T51193) (owner: 10Spage) [22:44:14] (03PS2) 10Reedy: Use ProfilerSectionOnly to handle DB/filebackend entries and the like [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183545 (owner: 10Aaron Schulz) [22:44:21] 3operations: migrate all ops-core items into operations project - https://phabricator.wikimedia.org/T87469#991985 (10RobH) 3NEW a:3chasemp [22:44:42] (03PS1) 10Alexandros Kosiaris: Introduce citoid.{eqiad,}.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/186516 [22:44:53] (03PS3) 10Dzahn: puppetize SSH private key for jenkins VE sync [puppet] - 10https://gerrit.wikimedia.org/r/186515 (https://phabricator.wikimedia.org/T84731) [22:44:55] (03CR) 10Jforrester: "Done in If98a6c8a6 instead." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/170129 (https://phabricator.wikimedia.org/T51193) (owner: 10Spage) [22:45:04] (03PS1) 10Ottomata: Fix dependency, fix template variables [puppet] - 10https://gerrit.wikimedia.org/r/186517 [22:45:15] (03CR) 10Jforrester: "This didn't remove the special-case for Labs…" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/175565 (owner: 10EBernhardson) [22:45:45] (03CR) 10jenkins-bot: [V: 04-1] Fix dependency, fix template variables [puppet] - 10https://gerrit.wikimedia.org/r/186517 (owner: 10Ottomata) [22:46:06] (03PS2) 10Alexandros Kosiaris: Introduce citoid.{eqiad,}.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/186516 [22:46:35] (03PS2) 10Ottomata: Fix dependency, fix template variables [puppet] - 10https://gerrit.wikimedia.org/r/186517 [22:47:03] (03PS3) 10Ottomata: Fix dependency, fix template variables [puppet] - 10https://gerrit.wikimedia.org/r/186517 [22:47:55] (03PS2) 10Reedy: Set wgCategoryCollation to 'uca-pl' on plwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185426 (https://phabricator.wikimedia.org/T86821) (owner: 10Glaisher) [22:48:05] (03CR) 10Reedy: [C: 032] Set wgCategoryCollation to 'uca-pl' on plwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185426 (https://phabricator.wikimedia.org/T86821) (owner: 10Glaisher) [22:48:09] (03Merged) 10jenkins-bot: Set wgCategoryCollation to 'uca-pl' on plwikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185426 (https://phabricator.wikimedia.org/T86821) (owner: 10Glaisher) [22:48:27] (03PS3) 10Alexandros Kosiaris: Introduce citoid.{eqiad,}.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/186516 [22:48:33] !log reedy Synchronized wmf-config/InitialiseSettings.php: plwikisource collation update (duration: 00m 06s) [22:48:39] Logged the message, Master [22:49:27] !log running mwscript updateCollation.php --wiki=plwikisource --previous-collation=uppercase in screen as reedy on tin [22:49:32] Logged the message, Master [22:50:08] 3Phabricator, operations: have any task put into ops-access-requests automatically generate an ops-access-review task - https://phabricator.wikimedia.org/T87467#992004 (10RobH) [22:50:42] (03CR) 10Ottomata: [C: 032] Fix dependency, fix template variables [puppet] - 10https://gerrit.wikimedia.org/r/186517 (owner: 10Ottomata) [22:51:39] (03CR) 10Alexandros Kosiaris: [C: 032] puppetize SSH private key for jenkins VE sync [puppet] - 10https://gerrit.wikimedia.org/r/186515 (https://phabricator.wikimedia.org/T84731) (owner: 10Dzahn) [22:52:01] !log restarting graphite on tungsten, webapp stuck and no graphs [22:52:07] Logged the message, Master [22:53:07] 3Phabricator, operations: have any task put into ops-access-requests automatically generate an ops-access-review task - https://phabricator.wikimedia.org/T87467#992011 (10mmodell) I thought this was already done as part of the security extension. Not extensively tested though. [22:54:11] (03PS1) 10Ottomata: require statistics::web in statistics::sites classes [puppet] - 10https://gerrit.wikimedia.org/r/186518 [22:54:33] (03PS2) 10Ottomata: require statistics::web in statistics::sites classes [puppet] - 10https://gerrit.wikimedia.org/r/186518 [22:55:10] Feel free to ping me re: ops-access-request subtasks, I'm the one who implemented the code for this and it should be working, or easily fixable if something's broken [22:55:20] (03CR) 10Ottomata: [C: 032] require statistics::web in statistics::sites classes [puppet] - 10https://gerrit.wikimedia.org/r/186518 (owner: 10Ottomata) [22:55:31] twentyafterfour: i assigned the ticket to you ;D [22:55:35] (03PS1) 10Ori.livneh: Route all counter stats to tungsten:3811 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186519 [22:55:42] (03CR) 10Dzahn: "Notice: /Stage[main]/Role::Ci::Master/File[/var/lib/jenkins/.ssh/jenkins-mwext-sync_id_rsa]/mode: mode changed '0600' to '0400'" [puppet] - 10https://gerrit.wikimedia.org/r/186515 (https://phabricator.wikimedia.org/T84731) (owner: 10Dzahn) [22:55:51] (03CR) 10Ori.livneh: [C: 032] Route all counter stats to tungsten:3811 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186519 (owner: 10Ori.livneh) [22:56:05] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 525 bytes in 0.018 second response time [22:56:36] RECOVERY - puppet last run on stat1001 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [22:56:50] (03Merged) 10jenkins-bot: Route all counter stats to tungsten:3811 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186519 (owner: 10Ori.livneh) [22:57:04] !log updateCollation.php for plwikisource complete [22:57:10] Logged the message, Master [22:57:17] (03PS1) 10Ottomata: Fix document root for geowiki-private data in stats.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/186520 [22:57:22] !log ori Synchronized wmf-config/CommonSettings.php: I50931db37: Route all counter stats to tungsten:3811 (duration: 00m 07s) [22:57:25] Logged the message, Master [22:58:26] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.018 second response time [22:58:47] (03CR) 10Ottomata: [C: 032] Fix document root for geowiki-private data in stats.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/186520 (owner: 10Ottomata) [22:58:54] 3Phabricator, operations: have any task put into ops-access-requests automatically generate an ops-access-review task - https://phabricator.wikimedia.org/T87467#992016 (10mmodell) The described behaviour is essentially how it works, however, it's not based on project tags, it's based on the security dropdown. It... [22:59:29] (03PS3) 10Reedy: Beta Features: Remove the (now default) New Search feature [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185076 (owner: 10Amire80) [22:59:47] 3Phabricator, operations: have any task put into ops-access-requests automatically generate an ops-access-review task - https://phabricator.wikimedia.org/T87467#992017 (10mmodell) Ah ha, I see the access request dropdown option is missing. @chasemp: any reason this isn't included in the config? [22:59:49] (03CR) 10Reedy: [C: 032] "Do we have a maintenance script or anything to run to clean up applicable database tables?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185076 (owner: 10Amire80) [22:59:53] (03Merged) 10jenkins-bot: Beta Features: Remove the (now default) New Search feature [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185076 (owner: 10Amire80) [23:00:06] (03PS4) 10Reedy: Beta Features: Disable the Compact Personal Bar feature [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185116 (https://phabricator.wikimedia.org/T85541) (owner: 10Jforrester) [23:00:12] (03CR) 10Reedy: [C: 032] Beta Features: Disable the Compact Personal Bar feature [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185116 (https://phabricator.wikimedia.org/T85541) (owner: 10Jforrester) [23:00:17] (03Merged) 10jenkins-bot: Beta Features: Disable the Compact Personal Bar feature [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185116 (https://phabricator.wikimedia.org/T85541) (owner: 10Jforrester) [23:00:18] 3Scrum-of-Scrums, RESTBase, operations, Services: Restbase deployment - https://phabricator.wikimedia.org/T1228#992019 (10mobrovac) [23:00:31] 3Phabricator, operations: have any task put into ops-access-requests automatically generate an ops-access-review task - https://phabricator.wikimedia.org/T87467#992024 (10chasemp) @mmodell, before today we weren't ready for it, and I wasn't sure if the logic was ported to Security-Extension-2.0? [23:00:37] (03PS1) 10KartikMistry: WIP: cxserver: enable no->nn language pair [puppet] - 10https://gerrit.wikimedia.org/r/186522 [23:01:04] (03PS3) 10Reedy: Use ProfilerSectionOnly to handle DB/filebackend entries and the like [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183545 (owner: 10Aaron Schulz) [23:01:07] (03CR) 10Reedy: [C: 032] Use ProfilerSectionOnly to handle DB/filebackend entries and the like [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183545 (owner: 10Aaron Schulz) [23:01:12] (03Merged) 10jenkins-bot: Use ProfilerSectionOnly to handle DB/filebackend entries and the like [mediawiki-config] - 10https://gerrit.wikimedia.org/r/183545 (owner: 10Aaron Schulz) [23:01:20] 3Phabricator, operations: have any task put into ops-access-requests automatically generate an ops-access-review task - https://phabricator.wikimedia.org/T87467#992026 (10mmodell) @chasemp the logic is ported, just not very well tested. I'll do some testing on my local phab and we can go from there. [23:01:48] (03PS1) 10Ottomata: Remove now unused parts of misc/statistics.pp [puppet] - 10https://gerrit.wikimedia.org/r/186523 [23:02:00] 3Phabricator, operations: have any task put into ops-access-requests automatically generate an ops-access-review task - https://phabricator.wikimedia.org/T87467#992027 (10chasemp) >>! In T87467#992026, @mmodell wrote: > @chasemp the logic is ported, just not very well tested. I'll do some testing on my local pha... [23:02:08] (03PS2) 10Reedy: Allow sysops/botadmins to grant/revoke 'confirmed' user group on fawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186230 (https://phabricator.wikimedia.org/T87348) (owner: 10Calak) [23:02:12] (03CR) 10Reedy: [C: 032] Allow sysops/botadmins to grant/revoke 'confirmed' user group on fawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186230 (https://phabricator.wikimedia.org/T87348) (owner: 10Calak) [23:02:17] (03Merged) 10jenkins-bot: Allow sysops/botadmins to grant/revoke 'confirmed' user group on fawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186230 (https://phabricator.wikimedia.org/T87348) (owner: 10Calak) [23:02:43] (03PS2) 10Ottomata: Remove now unused parts of misc/statistics.pp [puppet] - 10https://gerrit.wikimedia.org/r/186523 [23:02:53] (03PS2) 10Reedy: Add 'move-categorypages' right to ruwiki bots [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186236 (https://phabricator.wikimedia.org/T87230) (owner: 10Glaisher) [23:03:27] (03CR) 10Ottomata: [C: 032 V: 032] Remove now unused parts of misc/statistics.pp [puppet] - 10https://gerrit.wikimedia.org/r/186523 (owner: 10Ottomata) [23:03:46] (03CR) 10Reedy: [C: 032] Add 'move-categorypages' right to ruwiki bots [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186236 (https://phabricator.wikimedia.org/T87230) (owner: 10Glaisher) [23:04:24] (03Merged) 10jenkins-bot: Add 'move-categorypages' right to ruwiki bots [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186236 (https://phabricator.wikimedia.org/T87230) (owner: 10Glaisher) [23:06:16] (03PS1) 10Yuvipanda: tools: Install rlwrap in dev_environ [puppet] - 10https://gerrit.wikimedia.org/r/186525 (https://phabricator.wikimedia.org/T87368) [23:06:23] (03CR) 10jenkins-bot: [V: 04-1] tools: Install rlwrap in dev_environ [puppet] - 10https://gerrit.wikimedia.org/r/186525 (https://phabricator.wikimedia.org/T87368) (owner: 10Yuvipanda) [23:06:49] andrewbogott: thoughts on the DNS issues? [23:06:55] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: puppet fail [23:07:16] andrewbogott: where does labs instances resolv.conf come from? [23:07:25] (03PS1) 10Ottomata: Fixes for a few variable names now that they are in the statistics module [puppet] - 10https://gerrit.wikimedia.org/r/186526 [23:07:35] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: puppet fail [23:07:37] (03PS2) 10Reedy: Decrease '$wgAbuseFilterAnonBlockDuration' to '7 days' on fawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186156 (https://phabricator.wikimedia.org/T87317) (owner: 10Se4598) [23:07:43] andrewbogott: I see that we have postinst that’s removes resolv.conf [23:07:43] (03CR) 10Reedy: [C: 032] Decrease '$wgAbuseFilterAnonBlockDuration' to '7 days' on fawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186156 (https://phabricator.wikimedia.org/T87317) (owner: 10Se4598) [23:07:47] (03Merged) 10jenkins-bot: Decrease '$wgAbuseFilterAnonBlockDuration' to '7 days' on fawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186156 (https://phabricator.wikimedia.org/T87317) (owner: 10Se4598) [23:07:49] andrewbogott: and the puppet resolv.conf isn’t set for labs [23:07:55] YuviPanda: modules/base/templates/resolv.conf.erb if base is applied [23:08:02] mutante: nope, taht’s realm branched [23:08:07] i see [23:08:50] (03PS2) 10Yuvipanda: tools: Install rlwrap in dev_environ [puppet] - 10https://gerrit.wikimedia.org/r/186525 (https://phabricator.wikimedia.org/T87368) [23:09:19] (03CR) 10Yuvipanda: [C: 032] tools: Install rlwrap in dev_environ [puppet] - 10https://gerrit.wikimedia.org/r/186525 (https://phabricator.wikimedia.org/T87368) (owner: 10Yuvipanda) [23:09:39] (03CR) 10Yuvipanda: [V: 032] tools: Install rlwrap in dev_environ [puppet] - 10https://gerrit.wikimedia.org/r/186525 (https://phabricator.wikimedia.org/T87368) (owner: 10Yuvipanda) [23:09:53] (03PS2) 10Ottomata: Fixes for a few variable names now that they are in the statistics module [puppet] - 10https://gerrit.wikimedia.org/r/186526 [23:10:00] YuviPanda: I think it must be created by cloud init. Do you find it to be wrong in labs instances? [23:10:01] (03PS3) 10Ottomata: Fixes for a few variable names now that they are in the statistics module [puppet] - 10https://gerrit.wikimedia.org/r/186526 [23:10:16] andrewbogott: no, I am wondering if we can make it point to some other DNS service that we set up [23:10:24] andrewbogott: that just recurses to dnsmasq for some things [23:10:47] oh -- yes, that should be possible. Although I don't know that resolv.conf is puppetized for existing instances. Let me see... [23:11:05] andrewbogott: it is puppetized only for prod [23:11:05] (03PS2) 10Reedy: Disable xff logs. Bloated and no one uses them [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185923 [23:11:15] PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: There are 6 unmerged changes in mediawiki_config (dir /srv/mediawiki-staging/). [23:11:20] (03CR) 10Reedy: [C: 032] Disable xff logs. Bloated and no one uses them [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185923 (owner: 10Reedy) [23:11:24] (03Merged) 10jenkins-bot: Disable xff logs. Bloated and no one uses them [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185923 (owner: 10Reedy) [23:11:52] andrewbogott: maybe even have dnsmasq listen only locally, and have powerdns or something else (even bind?) serve labs... [23:12:31] YuviPanda: are our DNS problems internal to labs instances? Or is it visibility of instance names to the outside world that fails? [23:12:38] (03PS1) 10Tim Starling: /home/tstarling/.profile [puppet] - 10https://gerrit.wikimedia.org/r/186529 [23:12:40] andrewbogott: internal to labs instances [23:12:50] hm [23:13:05] (03CR) 10Ottomata: [C: 032] Fixes for a few variable names now that they are in the statistics module [puppet] - 10https://gerrit.wikimedia.org/r/186526 (owner: 10Ottomata) [23:13:05] 3Wikimedia-DNS, operations, WMF-Legal: wikibook.org should be hosted by WMF and should redirect to wikibooks.org - https://phabricator.wikimedia.org/T87375#992053 (10Dzahn) [23:13:22] (03CR) 10Tim Starling: [C: 032] /home/tstarling/.profile [puppet] - 10https://gerrit.wikimedia.org/r/186529 (owner: 10Tim Starling) [23:13:38] (03PS2) 10Filippo Giunchedi: lsearchd: reclaim search hosts [puppet] - 10https://gerrit.wikimedia.org/r/186511 (https://phabricator.wikimedia.org/T86149) [23:13:45] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] lsearchd: reclaim search hosts [puppet] - 10https://gerrit.wikimedia.org/r/186511 (https://phabricator.wikimedia.org/T86149) (owner: 10Filippo Giunchedi) [23:13:56] (03CR) 10Greg Grossmeier: "Yeah, we (Tony, Jeff and I) were chating in email on timing. They'll try to get it done on Monday/Tuesday." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186242 (owner: 1001tonythomas) [23:14:17] ottomata: good to puppet-merge ? [23:14:57] ja sorry [23:14:57] just did [23:16:06] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:16:17] oh you can puppet-merge a single commit? [23:16:55] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [23:17:05] PROBLEM - puppet last run on analytics1018 is CRITICAL: CRITICAL: Puppet has 1 failures [23:17:05] PROBLEM - puppet last run on wtp1001 is CRITICAL: CRITICAL: Puppet has 1 failures [23:17:16] PROBLEM - puppet last run on db1035 is CRITICAL: CRITICAL: Puppet has 1 failures [23:17:36] PROBLEM - puppet last run on mw1071 is CRITICAL: CRITICAL: Puppet has 1 failures [23:17:45] PROBLEM - puppet last run on es2010 is CRITICAL: CRITICAL: Puppet has 1 failures [23:18:16] PROBLEM - puppet last run on mw1064 is CRITICAL: CRITICAL: Puppet has 1 failures [23:18:26] PROBLEM - puppet last run on dysprosium is CRITICAL: CRITICAL: Puppet has 1 failures [23:18:36] PROBLEM - puppet last run on db1037 is CRITICAL: CRITICAL: Puppet has 1 failures [23:18:55] PROBLEM - puppet last run on mw1207 is CRITICAL: CRITICAL: Puppet has 1 failures [23:18:56] PROBLEM - puppet last run on mw1253 is CRITICAL: CRITICAL: Puppet has 1 failures [23:19:15] PROBLEM - puppet last run on mc1015 is CRITICAL: CRITICAL: Puppet has 1 failures [23:19:30] (looking) [23:20:15] PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: Puppet has 1 failures [23:20:42] TimStarling: ^ Error 404 on SERVER: Not Found: Could not find file_content modules/admin/home/tstarling/.profile [23:23:36] RECOVERY - puppet last run on helium is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [23:23:53] <_joe_> !log repooling 10 appservers depooled for no apparent reason, and no entry in the SAL [23:23:58] Logged the message, Master [23:25:54] (03PS1) 10Filippo Giunchedi: decomission lsearchd hosts [dns] - 10https://gerrit.wikimedia.org/r/186530 (https://phabricator.wikimedia.org/T86149) [23:26:03] (03PS1) 10Ottomata: Move misc::statistics::limn::data* to statistics module [puppet] - 10https://gerrit.wikimedia.org/r/186531 [23:26:09] (03CR) 10jenkins-bot: [V: 04-1] Move misc::statistics::limn::data* to statistics module [puppet] - 10https://gerrit.wikimedia.org/r/186531 (owner: 10Ottomata) [23:26:16] (03PS2) 10Ottomata: Move misc::statistics::limn::data* to statistics module [puppet] - 10https://gerrit.wikimedia.org/r/186531 [23:27:56] (03PS3) 10Ottomata: Move misc::statistics::limn::data* to statistics module [puppet] - 10https://gerrit.wikimedia.org/r/186531 [23:28:10] (03CR) 10Ottomata: [C: 032 V: 032] Move misc::statistics::limn::data* to statistics module [puppet] - 10https://gerrit.wikimedia.org/r/186531 (owner: 10Ottomata) [23:29:13] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] decomission lsearchd hosts [dns] - 10https://gerrit.wikimedia.org/r/186530 (https://phabricator.wikimedia.org/T86149) (owner: 10Filippo Giunchedi) [23:34:01] 3operations: reclaim lsearchd hosts - https://phabricator.wikimedia.org/T86149#992223 (10fgiunchedi) * removed from puppet / install-server * puppet stored config removed * puppet and salt keys removed * dns entries removed * machines halted [23:34:21] !log halting lsearchd machines [23:34:26] Logged the message, Master [23:34:45] (03PS1) 10Ottomata: Move cron_blog_pageviews to statistics module [puppet] - 10https://gerrit.wikimedia.org/r/186532 [23:34:46] RECOVERY - puppet last run on analytics1018 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [23:34:49] *curtain falls* [23:34:52] *applause* [23:34:55] RECOVERY - puppet last run on mw1064 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [23:34:56] RECOVERY - puppet last run on dysprosium is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [23:35:05] RECOVERY - puppet last run on db1035 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [23:35:16] RECOVERY - puppet last run on mw1071 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [23:35:27] RECOVERY - puppet last run on mw1253 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [23:35:27] RECOVERY - puppet last run on es2010 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [23:35:36] clap clap clap [23:35:44] (03CR) 10Ottomata: [C: 032] Move cron_blog_pageviews to statistics module [puppet] - 10https://gerrit.wikimedia.org/r/186532 (owner: 10Ottomata) [23:35:46] uuh [23:35:56] RECOVERY - puppet last run on wtp1001 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:36:16] RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [23:36:25] PROBLEM - Host search1008 is DOWN: PING CRITICAL - Packet loss = 100% [23:36:25] PROBLEM - Host search1009 is DOWN: PING CRITICAL - Packet loss = 100% [23:36:35] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [23:36:55] RECOVERY - puppet last run on mc1015 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:36:56] PROBLEM - Disk space on search1022 is CRITICAL: Connection refused by host [23:37:03] (03PS1) 10RobH: virt0.wikimedia.org cert expired [puppet] - 10https://gerrit.wikimedia.org/r/186534 [23:37:34] (03PS1) 10Ottomata: Remove include of misc::statisics::cron_blog_pageviews from site.pp [puppet] - 10https://gerrit.wikimedia.org/r/186535 [23:37:35] PROBLEM - puppet last run on search1023 is CRITICAL: Timeout while attempting connection [23:37:53] (03CR) 10Ottomata: [C: 032 V: 032] Remove include of misc::statisics::cron_blog_pageviews from site.pp [puppet] - 10https://gerrit.wikimedia.org/r/186535 (owner: 10Ottomata) [23:37:54] 3Phabricator, operations: have any task put into ops-access-requests automatically generate an ops-access-review task - https://phabricator.wikimedia.org/T87467#992259 (10mmodell) https://gerrit.wikimedia.org/r/#/c/186533/ [23:37:56] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [23:37:56] PROBLEM - Host search1014 is DOWN: PING CRITICAL - Packet loss = 100% [23:38:16] PROBLEM - Host search1015 is DOWN: PING CRITICAL - Packet loss = 100% [23:38:35] (03CR) 10RobH: [C: 032] virt0.wikimedia.org cert expired [puppet] - 10https://gerrit.wikimedia.org/r/186534 (owner: 10RobH) [23:38:46] (03CR) 10Alexandros Kosiaris: [C: 032] Re-use Parsoid Varnishes for citoid too [puppet] - 10https://gerrit.wikimedia.org/r/184904 (https://phabricator.wikimedia.org/T76949) (owner: 10Jforrester) [23:38:46] PROBLEM - Host search1016 is DOWN: PING CRITICAL - Packet loss = 100% [23:38:47] PROBLEM - Host search1018 is DOWN: PING CRITICAL - Packet loss = 100% [23:38:47] PROBLEM - Host search1019 is DOWN: PING CRITICAL - Packet loss = 100% [23:38:47] PROBLEM - Host search1017 is DOWN: PING CRITICAL - Packet loss = 100% [23:38:56] PROBLEM - Host search1021 is DOWN: PING CRITICAL - Packet loss = 100% [23:38:56] PROBLEM - Host search1020 is DOWN: PING CRITICAL - Packet loss = 100% [23:39:06] PROBLEM - Host search1024 is DOWN: PING CRITICAL - Packet loss = 100% [23:39:06] PROBLEM - Host searchidx1001 is DOWN: PING CRITICAL - Packet loss = 100% [23:39:06] PROBLEM - Host search1023 is DOWN: PING CRITICAL - Packet loss = 100% [23:39:34] someone forgot to remove them from the puppetstoreddb ;D [23:39:35] PROBLEM - Host search1022 is DOWN: PING CRITICAL - Packet loss = 100% [23:39:45] i did this to holmium earlier ;P [23:39:45] (03CR) 10Alexandros Kosiaris: [C: 032] Introduce citoid.{eqiad,}.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/186516 (owner: 10Alexandros Kosiaris) [23:40:12] 3Phabricator, operations: have any task put into ops-access-requests automatically generate an ops-access-review task - https://phabricator.wikimedia.org/T87467#992273 (10mmodell) @chasemp: See my patch, I fixed a couple of little things (fallout from upstream changes) and this seems to be working well now. [23:40:47] robh: yeah I removed them but forgot to disable puppet so they got re-added (I think) ;_; [23:41:31] 3Phabricator, operations: have any task put into ops-access-requests automatically generate an ops-access-review task - https://phabricator.wikimedia.org/T87467#992276 (10mmodell) One thing you might not want - it currently includes the "CC'd can see this" behaviour... [23:41:32] thats what i did [23:41:36] thats exactly what i did, heh [23:41:42] (03PS4) 10Reedy: Beta: Fix spacing and indentation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178772 (owner: 10KartikMistry) [23:41:42] RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: No changes to merge. [23:41:50] (03CR) 10Reedy: [C: 032] Beta: Fix spacing and indentation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178772 (owner: 10KartikMistry) [23:41:52] and it called in on me in the fwe minutes from removing and shutdown [23:41:54] (03Merged) 10jenkins-bot: Beta: Fix spacing and indentation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178772 (owner: 10KartikMistry) [23:42:18] yeah, many stepping stones [23:42:44] robh: ops-access-requests should be good once my patch is merged and the option is added to the security dropdown [23:43:00] !log reedy Synchronized wmf-config/: Config updates (duration: 00m 06s) [23:43:10] Logged the message, Master [23:43:31] (03PS2) 10Reedy: Add *.bnf.fr to the wgCopyUploadsDomains whitelist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184872 (https://phabricator.wikimedia.org/T86699) (owner: 10Dan-nl) [23:43:47] robh: I think I'm done with shutting lsearchd machines in https://phabricator.wikimedia.org/T86149 last step is reassing to you for wiping (?) [23:44:04] (03CR) 10Reedy: [C: 032] Add *.bnf.fr to the wgCopyUploadsDomains whitelist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184872 (https://phabricator.wikimedia.org/T86699) (owner: 10Dan-nl) [23:44:08] (03Merged) 10jenkins-bot: Add *.bnf.fr to the wgCopyUploadsDomains whitelist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184872 (https://phabricator.wikimedia.org/T86699) (owner: 10Dan-nl) [23:44:35] (03CR) 10Reedy: [C: 04-1] "Needs rebasing" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185155 (owner: 10Amire80) [23:44:49] (03PS2) 10Reedy: Add 'movefile' right to 'autopatrolled' group on frwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185997 (https://phabricator.wikimedia.org/T87145) (owner: 10Glaisher) [23:45:06] (03CR) 10Reedy: [C: 032] Add 'movefile' right to 'autopatrolled' group on frwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185997 (https://phabricator.wikimedia.org/T87145) (owner: 10Glaisher) [23:45:11] (03Merged) 10jenkins-bot: Add 'movefile' right to 'autopatrolled' group on frwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185997 (https://phabricator.wikimedia.org/T87145) (owner: 10Glaisher) [23:45:42] (03PS2) 10Reedy: Add extra namespaces for fawikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186317 (https://phabricator.wikimedia.org/T76663) (owner: 10Mjbmr) [23:46:43] (03PS3) 10Reedy: Add extra namespaces for fawikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186317 (https://phabricator.wikimedia.org/T76663) (owner: 10Mjbmr) [23:47:06] (03CR) 10Reedy: [C: 032] "Please read https://www.mediawiki.org/wiki/Gerrit/Commit_message_guidelines" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186317 (https://phabricator.wikimedia.org/T76663) (owner: 10Mjbmr) [23:47:10] (03Merged) 10jenkins-bot: Add extra namespaces for fawikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186317 (https://phabricator.wikimedia.org/T76663) (owner: 10Mjbmr) [23:47:36] 3operations: reclaim lsearchd hosts - https://phabricator.wikimedia.org/T86149#992290 (10fgiunchedi) a:5fgiunchedi>3RobH moving to robh as per https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Reclaim_or_Decommission [23:48:48] (03PS2) 10Reedy: Changing logo for klwiki from $stdlogo to commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184121 (https://phabricator.wikimedia.org/T86313) (owner: 10Steinsplitter) [23:49:01] (03CR) 10Reedy: [C: 032] Changing logo for klwiki from $stdlogo to commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184121 (https://phabricator.wikimedia.org/T86313) (owner: 10Steinsplitter) [23:49:05] (03Merged) 10jenkins-bot: Changing logo for klwiki from $stdlogo to commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/184121 (https://phabricator.wikimedia.org/T86313) (owner: 10Steinsplitter) [23:49:16] so we keep getting tons of puppet failures in beta labs due to temporary name resolution failures... [23:49:20] (03PS1) 10KartikMistry: WIP: cxserver: Add Yandex MT support [puppet] - 10https://gerrit.wikimedia.org/r/186538 [23:49:43] (03PS2) 10Amire80: Consistent hyphen in beta features whitelist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185155 [23:49:48] it doesn't seem like I can access the dns server at 10.68.16.1 [23:50:01] (03PS1) 10Mattflaschen: Simplify Echo and Thanks settings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186539 [23:50:23] (03PS2) 10Reedy: Enable VisualEditor on 'Draft' (118) namespace at hewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185449 (https://phabricator.wikimedia.org/T87027) (owner: 10Glaisher) [23:50:32] (03CR) 10Reedy: [C: 032] Enable VisualEditor on 'Draft' (118) namespace at hewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185449 (https://phabricator.wikimedia.org/T87027) (owner: 10Glaisher) [23:50:37] (03Merged) 10jenkins-bot: Enable VisualEditor on 'Draft' (118) namespace at hewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185449 (https://phabricator.wikimedia.org/T87027) (owner: 10Glaisher) [23:50:56] (03PS3) 10Reedy: Consistent hyphen in beta features whitelist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185155 (owner: 10Amire80) [23:51:02] (03CR) 10Reedy: [C: 032] Consistent hyphen in beta features whitelist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185155 (owner: 10Amire80) [23:51:09] (03Merged) 10jenkins-bot: Consistent hyphen in beta features whitelist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185155 (owner: 10Amire80) [23:51:30] (03PS2) 10Reedy: Update client lists for test / wikidata change dispatching [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185439 (owner: 10Aude) [23:51:35] (03CR) 10Reedy: [C: 032] Update client lists for test / wikidata change dispatching [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185439 (owner: 10Aude) [23:51:40] (03Merged) 10jenkins-bot: Update client lists for test / wikidata change dispatching [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185439 (owner: 10Aude) [23:52:21] RoanKattouw: any chance you will have 5mins at some point ? [23:52:32] (03PS3) 10Reedy: Hygiene: Change wgMFAnonymousEditing to wgMFEditorOptions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182177 (owner: 10Florianschmidtwelzow) [23:52:33] akosiaris: Sure, I have 7 mins right now [23:52:40] (03CR) 10Reedy: [C: 032] Hygiene: Change wgMFAnonymousEditing to wgMFEditorOptions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182177 (owner: 10Florianschmidtwelzow) [23:52:44] (03Merged) 10jenkins-bot: Hygiene: Change wgMFAnonymousEditing to wgMFEditorOptions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/182177 (owner: 10Florianschmidtwelzow) [23:53:41] !log reedy Synchronized wmf-config/: More config updates (duration: 00m 06s) [23:53:51] Logged the message, Master [23:57:02] 3operations: Intermittent DNS failures in beta labs regularly trigger a bunch of puppet failures - https://phabricator.wikimedia.org/T87480#992326 (10mmodell) 3NEW [23:57:20] (03CR) 10Mjbmr: "Thank you, this is a case of namespaceDupes also." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186317 (https://phabricator.wikimedia.org/T76663) (owner: 10Mjbmr) [23:57:37] 3Beta-Cluster, Release-Engineering, operations: Intermittent DNS failures in beta labs regularly trigger a bunch of puppet failures - https://phabricator.wikimedia.org/T87480#992333 (10mmodell) [23:57:48] 3Beta-Cluster, operations: Intermittent DNS failures in beta labs regularly trigger a bunch of puppet failures - https://phabricator.wikimedia.org/T87480#992335 (10Reedy) [23:57:55] stupid phab