[00:36:56] PROBLEM - puppet last run on cp3011 is CRITICAL: CRITICAL: puppet fail [00:52:36] PROBLEM - MySQL Replication Heartbeat on db1016 is CRITICAL: CRIT replication delay 364 seconds [00:52:48] PROBLEM - MySQL Slave Delay on db1016 is CRITICAL: CRIT replication delay 368 seconds [00:53:37] RECOVERY - MySQL Replication Heartbeat on db1016 is OK: OK replication delay -0 seconds [00:53:46] RECOVERY - MySQL Slave Delay on db1016 is OK: OK replication delay 0 seconds [00:56:44] RECOVERY - puppet last run on cp3011 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [02:15:12] !log LocalisationUpdate completed (1.25wmf4) at 2014-10-26 02:15:12+00:00 [02:15:23] Logged the message, Master [02:25:47] !log LocalisationUpdate completed (1.25wmf5) at 2014-10-26 02:25:47+00:00 [02:25:54] Logged the message, Master [03:36:20] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Oct 26 03:36:20 UTC 2014 (duration 36m 19s) [03:36:27] Logged the message, Master [06:28:17] PROBLEM - puppet last run on search1007 is CRITICAL: CRITICAL: puppet fail [06:28:27] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: puppet fail [06:28:38] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: puppet fail [06:29:08] PROBLEM - puppet last run on amslvs1 is CRITICAL: CRITICAL: puppet fail [06:29:08] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: puppet fail [06:29:27] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:48] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:48] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:57] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:58] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:58] PROBLEM - puppet last run on dbproxy1001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:08] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:08] PROBLEM - puppet last run on mw1011 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:09] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:10] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:27] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:27] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:28] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:37] PROBLEM - puppet last run on ms-fe2001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:37] PROBLEM - puppet last run on amssq60 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:47] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:57] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:30] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:31] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:40] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:24] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:45:34] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:45:43] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:45:44] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:45:53] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:45:55] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:46:03] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:46:04] RECOVERY - puppet last run on ms-fe2001 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [06:46:14] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [06:46:14] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:46:23] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:46:23] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:46:33] RECOVERY - puppet last run on dbproxy1001 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:46:35] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:46:35] RECOVERY - puppet last run on mw1011 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:46:43] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:46:45] RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:46:45] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:46:45] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:46:53] RECOVERY - puppet last run on search1007 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:46:53] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:47:03] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:47:13] RECOVERY - puppet last run on amssq60 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:47:17] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:47:25] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [06:49:34] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Puppet has 1 failures [07:07:54] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:25:23] PROBLEM - puppet last run on achernar is CRITICAL: CRITICAL: Puppet has 1 failures [09:12:55] PROBLEM - puppet last run on mc1001 is CRITICAL: CRITICAL: Puppet has 1 failures [09:31:35] RECOVERY - puppet last run on mc1001 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [14:28:27] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: puppet fail [14:34:18] PROBLEM - puppet last run on amssq31 is CRITICAL: CRITICAL: puppet fail [14:46:59] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [14:54:57] RECOVERY - puppet last run on amssq31 is OK: OK: Puppet is currently enabled, last run 61 seconds ago with 0 failures [15:06:44] <_joe_> !log restarted hhvm on mw1114, memory nearly exhausted [15:06:52] Logged the message, Master [15:09:07] PROBLEM - puppet last run on install2001 is CRITICAL: CRITICAL: puppet fail [15:14:08] !log Jenkins/Zuul is stuck as of 20 hours ago [15:14:12] Logged the message, Master [15:15:07] RECOVERY - puppet last run on achernar is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [15:28:27] RECOVERY - puppet last run on install2001 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [18:05:55] RECOVERY - check if salt-minion is running on virt1006 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [18:08:55] PROBLEM - check if salt-minion is running on virt1006 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion [19:58:20] PROBLEM - puppet last run on cp3021 is CRITICAL: CRITICAL: puppet fail [20:16:57] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [21:22:28] (03PS1) 10Kaldari: Turning on $wgCopyUploadsFromSpecialUpload for Commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/168908 (https://bugzilla.wikimedia.org/71897) [21:45:06] PROBLEM - puppet last run on amssq32 is CRITICAL: CRITICAL: puppet fail [22:04:36] RECOVERY - puppet last run on amssq32 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [23:33:47] (03PS1) 10Nemo bis: Set $wgMFAnonymousEditing = true for Italian Wikipedia in November-December 2014 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/168915 (https://bugzilla.wikimedia.org/72541) [23:46:35] !log Force restarted Zuul [23:46:37] (03CR) 10Hoo man: Create new user group for WMDE staff at wikidatawiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/168771 (https://bugzilla.wikimedia.org/72459) (owner: 10Vogone) [23:46:46] Logged the message, Master [23:50:45] (03CR) 10Vogone: Create new user group for WMDE staff at wikidatawiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/168771 (https://bugzilla.wikimedia.org/72459) (owner: 10Vogone) [23:52:26] Krinkle: so...we have to comment recheck on everything now? [23:53:28] (and thanks!)