[00:00:25] PROBLEM - DPKG on analytics1026 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:00:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [00:01:25] RECOVERY - DPKG on analytics1026 is OK: All packages OK [00:06:15] PROBLEM - DPKG on analytics1014 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:07:15] RECOVERY - DPKG on analytics1014 is OK: All packages OK [00:23:15] RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000 [00:26:27] PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:52:56] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:53:45] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 2.170 second response time [02:15:07] !log LocalisationUpdate completed (1.22wmf16) at Fri Sep 13 02:15:07 UTC 2013 [02:15:12] Logged the message, Master [02:18:32] (03CR) 10MZMcBride: "A hasty decision is not served by a second hasty decision. Let's wait a few days to figure out how to best proceed. There may be actual bu" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84102 (owner: 10Ricordisamoa) [02:33:49] !log LocalisationUpdate completed (1.22wmf17) at Fri Sep 13 02:33:49 UTC 2013 [02:33:52] Logged the message, Master [02:54:31] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Sep 13 02:54:31 UTC 2013 [02:54:34] Logged the message, Master [03:25:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:26:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 7.615 second response time [03:34:19] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:36:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 4.639 second response time [03:43:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:45:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 6.800 second response time [03:51:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:52:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [04:01:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:02:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [04:22:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:23:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [04:32:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:33:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [05:33:52] RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000 [05:36:59] PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:02:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:06:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 5.578 second response time [08:13:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:18:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.135 second response time [08:38:42] (03PS1) 10Nemo bis: [CleanChanges] Set $wgCCTrailerFilter to true [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84113 [08:39:40] (03CR) 10Nemo bis: "Followup: https://gerrit.wikimedia.org/r/#/c/84113/" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81678 (owner: 10Nemo bis) [09:36:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.146 second response time [09:52:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:53:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time [10:36:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:38:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.136 second response time [10:47:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:48:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.136 second response time [11:31:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:32:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [12:57:40] (03PS1) 10Petr Onderka: Don't use the override keyword for GCC 4.6 [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84118 [12:59:32] (03Abandoned) 10Petr Onderka: Don't use the override keyword for GCC 4.6 [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84118 (owner: 10Petr Onderka) [14:10:21] (03Restored) 10Petr Onderka: Don't use the override keyword for GCC 4.6 [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84118 (owner: 10Petr Onderka) [14:10:32] (03PS2) 10Petr Onderka: The override keyword is not supported in GCC 4.6 [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84118 [14:10:33] (03PS1) 10Petr Onderka: GCC 4.6 requires --std=c++0x [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84123 [14:10:34] (03PS1) 10Petr Onderka: More fixes for GCC 4.6 [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84124 [14:11:12] (03CR) 10Petr Onderka: [C: 032 V: 032] GCC 4.6 requires --std=c++0x [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84123 (owner: 10Petr Onderka) [14:11:30] (03CR) 10Petr Onderka: [C: 032 V: 032] The override keyword is not supported in GCC 4.6 [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84118 (owner: 10Petr Onderka) [14:11:41] (03CR) 10Petr Onderka: [C: 032 V: 032] More fixes for GCC 4.6 [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84124 (owner: 10Petr Onderka) [14:21:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:24:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [14:25:58] hi Reedy [14:30:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.510 second response time [14:35:36] RECOVERY - check_job_queue on hume is OK: JOBQUEUE OK - all job queues below 10,000 [14:38:46] PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [14:39:08] (03PS1) 10Petr Onderka: LZMA doesn't require C++11 [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84126 [14:39:55] (03CR) 10Petr Onderka: [C: 032 V: 032] LZMA doesn't require C++11 [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84126 (owner: 10Petr Onderka) [14:46:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:39] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [15:01:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:02:37] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 1.374 second response time [15:17:03] (03PS1) 10Petr Onderka: The create action reads from stdin when file is - [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84127 [15:45:17] (03PS1) 10Petr Onderka: Fix for "Don't write empty indexes" [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84136 [15:48:52] (03PS2) 10Petr Onderka: Fix for "Don't write empty indexes" [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84136 [15:52:11] (03PS2) 10Petr Onderka: The create action reads from stdin when file is - [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84127 [15:52:12] (03PS3) 10Petr Onderka: Fix for "Don't write empty indexes" [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84136 [15:53:10] (03CR) 10Petr Onderka: [C: 032 V: 032] Fix for "Don't write empty indexes" [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84136 (owner: 10Petr Onderka) [15:53:22] (03CR) 10Petr Onderka: [C: 032 V: 032] The create action reads from stdin when file is - [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84127 (owner: 10Petr Onderka) [16:19:54] RECOVERY - check_job_queue on hume is OK: JOBQUEUE OK - all job queues below 10,000 [16:23:04] PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [17:48:17] !log testing incr dump code on snapshot1003 in screen; if it falls over it won't break anything [17:48:21] Logged the message, Master [18:47:34] (03CR) 10MZMcBride: "Who?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/78944 (owner: 10QChris) [19:07:46] PROBLEM - MySQL Processlist on db1009 is CRITICAL: CRIT 0 unauthenticated, 1 locked, 0 copy to table, 39 statistics [19:08:52] RECOVERY - MySQL Processlist on db1009 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 3 statistics [19:08:59] (03PS1) 10Petr Onderka: Another fix for "Don't write empty indexes" [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84141 [19:09:19] (03CR) 10Petr Onderka: [C: 032 V: 032] Another fix for "Don't write empty indexes" [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/84141 (owner: 10Petr Onderka) [20:21:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:22:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 7.165 second response time [20:29:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:33:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [20:57:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:05:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 6.474 second response time [21:22:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:24:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [23:22:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:23:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [23:44:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:45:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 1.079 second response time [23:51:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:53:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time