[00:03:50] (03PS1) 10BryanDavis: logstash: parse json encoded hhvm fatal errors [puppet] - 10https://gerrit.wikimedia.org/r/179759 [00:08:02] (03PS1) 10Andrew Bogott: Reprioritize partitioning so we get a swap. [puppet] - 10https://gerrit.wikimedia.org/r/179760 [00:11:25] (03PS2) 10BryanDavis: logstash: parse json encoded hhvm fatal errors [puppet] - 10https://gerrit.wikimedia.org/r/179759 [00:13:08] (03CR) 10Andrew Bogott: [C: 032] Reprioritize partitioning so we get a swap. [puppet] - 10https://gerrit.wikimedia.org/r/179760 (owner: 10Andrew Bogott) [00:17:45] (03CR) 10BryanDavis: "Tested in beta via cherry-pick to deployment-salt" [puppet] - 10https://gerrit.wikimedia.org/r/179759 (owner: 10BryanDavis) [00:40:21] (03PS3) 10BryanDavis: logstash: parse json encoded hhvm fatal errors [puppet] - 10https://gerrit.wikimedia.org/r/179759 [01:19:38] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: Puppet has 1 failures [01:21:37] (03PS4) 10BryanDavis: logstash: parse json encoded hhvm fatal errors [puppet] - 10https://gerrit.wikimedia.org/r/179759 [01:34:56] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:36:06] (03PS5) 10BryanDavis: logstash: parse json encoded hhvm fatal errors [puppet] - 10https://gerrit.wikimedia.org/r/179759 [01:37:12] (03PS1) 10GWicke: Improve restbase init [puppet] - 10https://gerrit.wikimedia.org/r/179764 [01:46:26] (03PS4) 10BryanDavis: logstash: Parse apache syslog messages [puppet] - 10https://gerrit.wikimedia.org/r/179480 [01:47:26] (03PS6) 10BryanDavis: logstash: parse json encoded hhvm fatal errors [puppet] - 10https://gerrit.wikimedia.org/r/179759 [01:59:04] ori: Did that change to StartProfiler not get synced? I'm seeing lots of errors from across the cluster still referencing a foreach on line 122. According to git the foreach is on line 124 now. [02:11:39] !log l10nupdate Synchronized php-1.25wmf11/cache/l10n: (no message) (duration: 00m 02s) [02:11:43] !log LocalisationUpdate completed (1.25wmf11) at 2014-12-14 02:11:43+00:00 [02:11:49] Logged the message, Master [02:11:53] Logged the message, Master [02:16:27] !log l10nupdate Synchronized php-1.25wmf12/cache/l10n: (no message) (duration: 00m 01s) [02:16:31] !log LocalisationUpdate completed (1.25wmf12) at 2014-12-14 02:16:31+00:00 [02:16:35] Logged the message, Master [02:16:41] Logged the message, Master [02:22:13] (03PS1) 10Andrew Bogott: Support bootstrap-vz for buildign labs debian images [puppet] - 10https://gerrit.wikimedia.org/r/179765 [03:11:52] PROBLEM - DPKG on hafnium is CRITICAL: DPKG CRITICAL dpkg reports broken packages [03:33:14] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Dec 14 03:33:14 UTC 2014 (duration 33m 13s) [03:33:19] Logged the message, Master [03:39:04] PROBLEM - puppet last run on mw1243 is CRITICAL: CRITICAL: Puppet has 1 failures [03:39:04] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: Puppet has 1 failures [03:39:04] PROBLEM - puppet last run on db1057 is CRITICAL: CRITICAL: Puppet has 2 failures [03:39:07] PROBLEM - puppet last run on mw1087 is CRITICAL: CRITICAL: Puppet has 1 failures [03:39:35] PROBLEM - puppet last run on db1064 is CRITICAL: CRITICAL: Puppet has 1 failures [03:39:51] PROBLEM - puppet last run on mw1032 is CRITICAL: CRITICAL: Puppet has 1 failures [03:41:52] PROBLEM - puppet last run on amssq62 is CRITICAL: CRITICAL: Puppet has 1 failures [03:42:17] PROBLEM - puppet last run on cp3012 is CRITICAL: CRITICAL: Puppet has 2 failures [03:52:05] PROBLEM - puppet last run on mc1006 is CRITICAL: CRITICAL: Puppet has 1 failures [03:53:22] RECOVERY - puppet last run on mw1243 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [03:53:25] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [03:53:41] RECOVERY - puppet last run on db1057 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:53:42] RECOVERY - puppet last run on amssq62 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [03:54:01] RECOVERY - puppet last run on cp3012 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [03:54:15] RECOVERY - puppet last run on mw1087 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:54:38] RECOVERY - puppet last run on db1064 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:55:00] RECOVERY - puppet last run on mw1032 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:00:51] PROBLEM - puppet last run on mw1165 is CRITICAL: CRITICAL: Puppet has 1 failures [04:04:01] RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [04:10:00] RECOVERY - puppet last run on mw1165 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:33:16] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 3 failures [06:35:05] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 2 failures [06:35:07] PROBLEM - puppet last run on db1023 is CRITICAL: CRITICAL: puppet fail [06:35:11] PROBLEM - puppet last run on analytics1030 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:56] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 3 failures [06:36:51] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 3 failures [06:42:52] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: Puppet has 2 failures [06:45:49] RECOVERY - puppet last run on analytics1030 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [06:46:33] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:46:52] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:47:12] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:49:08] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [06:49:08] RECOVERY - puppet last run on db1023 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:14:16] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:36:56] (03CR) 10Steinsplitter: "@Legoktm: what i know is that wmflabs is not affected by the proxy problem." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179094 (owner: 10Steinsplitter) [07:39:08] fyi, I'm getting DB outages intermittently on page reads on wikitech: "(Cannot contact the database server: Too many connections (208.80.154.18))" [08:19:38] PROBLEM - puppet last run on cp3009 is CRITICAL: CRITICAL: puppet fail [08:34:55] RECOVERY - puppet last run on cp3009 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:37:21] PROBLEM - puppet last run on mw1168 is CRITICAL: CRITICAL: Puppet has 1 failures [08:49:21] RECOVERY - puppet last run on mw1168 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [09:18:09] PROBLEM - puppet last run on mw1029 is CRITICAL: CRITICAL: Puppet has 1 failures [09:32:57] RECOVERY - puppet last run on mw1029 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:04:13] PROBLEM - puppet last run on xenon is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:13:28] (03PS2) 10Giuseppe Lavagetto: hhvm: remove jemalloc profiling completely [puppet] - 10https://gerrit.wikimedia.org/r/179609 [11:14:01] (03CR) 10Giuseppe Lavagetto: [C: 032] hhvm: remove jemalloc profiling completely [puppet] - 10https://gerrit.wikimedia.org/r/179609 (owner: 10Giuseppe Lavagetto) [11:16:11] <_joe_> jenkins is able to make me wait even on sunday [11:17:24] _joe_: it is a hint you shouldn't be working [11:18:22] <_joe_> matanya: I'm volunteering! [11:18:39] heh, nice one [13:04:35] (03PS1) 10Andrew Bogott: Yet another preseed attempt [puppet] - 10https://gerrit.wikimedia.org/r/179781 [13:05:43] (03CR) 10Andrew Bogott: [C: 032] Yet another preseed attempt [puppet] - 10https://gerrit.wikimedia.org/r/179781 (owner: 10Andrew Bogott) [14:54:57] (03CR) 10JanZerebecki: [C: 031] Update entity suggester blacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179469 (owner: 10Hoo man) [18:16:05] (03CR) 10Thiemo Mättig (WMDE): [C: 04-1] Don't collapse sections on mobile WD (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179513 (owner: 10MaxSem) [18:30:22] (03CR) 10Thiemo Mättig (WMDE): [C: 031] "Double checked, every removal/addition is correct." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179469 (owner: 10Hoo man) [19:13:50] (03PS1) 10Ori.livneh: Add 'xenon' module for aggregating ext_xenon-produced traces [puppet] - 10https://gerrit.wikimedia.org/r/179791 [19:14:36] (03CR) 10jenkins-bot: [V: 04-1] Add 'xenon' module for aggregating ext_xenon-produced traces [puppet] - 10https://gerrit.wikimedia.org/r/179791 (owner: 10Ori.livneh) [19:26:36] <_joe_> andrewbogott_afk: whever you're here, could you move the VMs hhvm-img and lamp-img so that they are on the same physical host? [20:50:55] PROBLEM - nutcracker port on mw1202 is CRITICAL: Cannot assign requested address [20:54:00] RECOVERY - nutcracker port on mw1202 is OK: TCP OK - 0.000 second response time on port 11212 [21:42:34] PROBLEM - puppet last run on db1035 is CRITICAL: CRITICAL: puppet fail [21:57:38] RECOVERY - puppet last run on db1035 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:11:48] (03PS2) 10Ori.livneh: Add 'xenon' module for aggregating ext_xenon-produced traces [puppet] - 10https://gerrit.wikimedia.org/r/179791 [23:14:11] --mindwidth=1 should make it a bit better