[00:01:47] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [00:08:06] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [01:10:10] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [01:31:50] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset -7.021427155e-05 secs [01:32:40] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset -0.0007178783417 secs [02:03:42] !log LocalisationUpdate completed (1.22wmf8) at Sat Jun 29 02:03:41 UTC 2013 [02:03:52] Logged the message, Master [02:06:09] !log LocalisationUpdate completed (1.22wmf9) at Sat Jun 29 02:06:08 UTC 2013 [02:06:17] Logged the message, Master [02:09:26] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [02:11:56] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Jun 29 02:11:56 UTC 2013 [02:12:05] Logged the message, Master [02:52:07] PROBLEM - Puppet freshness on wtp1015 is CRITICAL: No successful Puppet run in the last 10 hours [02:59:07] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [03:06:07] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: No successful Puppet run in the last 10 hours [03:09:39] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [04:01:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:02:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [04:09:38] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [04:27:47] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:29:37] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [05:09:57] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [05:13:47] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:14:37] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [05:38:39] PROBLEM - Puppet freshness on mc1005 is CRITICAL: No successful Puppet run in the last 10 hours [05:38:39] PROBLEM - Puppet freshness on mw41 is CRITICAL: No successful Puppet run in the last 10 hours [05:39:39] PROBLEM - Puppet freshness on db1036 is CRITICAL: No successful Puppet run in the last 10 hours [05:39:39] PROBLEM - Puppet freshness on amssq44 is CRITICAL: No successful Puppet run in the last 10 hours [05:40:39] PROBLEM - Puppet freshness on ms-be3 is CRITICAL: No successful Puppet run in the last 10 hours [05:41:39] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [05:41:39] PROBLEM - Puppet freshness on mw58 is CRITICAL: No successful Puppet run in the last 10 hours [05:41:39] PROBLEM - Puppet freshness on sq42 is CRITICAL: No successful Puppet run in the last 10 hours [05:41:49] RECOVERY - search indices - check lucene status page on search1007 is OK: HTTP OK: HTTP/1.1 200 OK - 351 bytes in 0.002 second response time [05:41:49] RECOVERY - search indices - check lucene status page on search1008 is OK: HTTP OK: HTTP/1.1 200 OK - 351 bytes in 0.004 second response time [05:45:59] PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server [05:45:59] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server [06:07:19] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [06:08:17] PROBLEM - Puppet freshness on aluminium is CRITICAL: No successful Puppet run in the last 10 hours [06:08:17] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:17] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:17] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: No successful Puppet run in the last 10 hours [06:08:17] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: No successful Puppet run in the last 10 hours [06:10:17] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: No successful Puppet run in the last 10 hours [06:10:17] PROBLEM - Puppet freshness on analytics1023 is CRITICAL: No successful Puppet run in the last 10 hours [06:10:17] PROBLEM - Puppet freshness on db58 is CRITICAL: No successful Puppet run in the last 10 hours [06:10:17] PROBLEM - Puppet freshness on db1029 is CRITICAL: No successful Puppet run in the last 10 hours [06:10:17] PROBLEM - Puppet freshness on ms-be1012 is CRITICAL: No successful Puppet run in the last 10 hours [06:30:27] RECOVERY - Disk space on ms-be1002 is OK: DISK OK [06:31:17] RECOVERY - Disk space on ms-be1001 is OK: DISK OK [06:32:16] New patchset: Ori.livneh; "Set common rsync and dsh parameters in mw-deployment-vars" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57890 [06:38:49] New review: Ori.livneh; "Tim, I've implemented your suggestions from May 14." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/57890 [07:05:13] anyone else having trouble connecting to the cluster wikis? [07:07:02] hmm, can't seem to connect to the labs servers either [07:08:05] PING wikipedia-lb.eqiad.wikimedia.org (208.80.154.225): 56 data bytes [07:08:06] Request timeout for icmp_seq 0 [07:08:07] etc. [07:10:29] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [07:11:00] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [07:11:00] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [07:11:00] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [07:11:00] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [07:11:01] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [07:11:01] PROBLEM - Puppet freshness on sodium is CRITICAL: No successful Puppet run in the last 10 hours [07:11:01] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [07:11:02] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [07:11:02] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [07:11:52] never mind, seems to be working again [07:46:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:47:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [08:10:05] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [08:56:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:57:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.143 second response time [09:02:33] RECOVERY - NTP on ssl3003 is OK: NTP OK: Offset -0.0001499652863 secs [09:02:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:03:03] RECOVERY - NTP on ssl3002 is OK: NTP OK: Offset -0.00395488739 secs [09:03:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [09:07:59] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [09:08:41] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [09:31:42] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:32:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [10:08:09] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [11:30:47] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:31:47] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 6.821 second response time [12:08:35] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [12:16:44] PROBLEM - Disk space on ms-be1001 is CRITICAL: DISK CRITICAL - free space: / 5698 MB (3% inode=98%): [12:52:11] PROBLEM - Puppet freshness on wtp1015 is CRITICAL: No successful Puppet run in the last 10 hours [12:59:11] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [13:06:11] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: No successful Puppet run in the last 10 hours [13:07:28] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [13:34:49] RECOVERY - Disk space on ms-be1001 is OK: DISK OK [14:09:50] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [14:29:01] anyone in the ldap/ops group around? [14:29:15] minor gerrit administrative change that's available only to that group, help would be appreciated :) [14:29:48] New review: JanZerebecki; "How about using an internal rewrite instead of an external redirect? For example use the [PT] flag a..." [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/65443 [15:07:49] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [15:38:50] PROBLEM - Puppet freshness on mc1005 is CRITICAL: No successful Puppet run in the last 10 hours [15:38:50] PROBLEM - Puppet freshness on mw41 is CRITICAL: No successful Puppet run in the last 10 hours [15:38:56] !log updated Parsoid to 9fad5fa [15:39:48] PROBLEM - Puppet freshness on amssq44 is CRITICAL: No successful Puppet run in the last 10 hours [15:39:48] PROBLEM - Puppet freshness on db1036 is CRITICAL: No successful Puppet run in the last 10 hours [15:40:48] PROBLEM - Puppet freshness on ms-be3 is CRITICAL: No successful Puppet run in the last 10 hours [15:41:48] PROBLEM - Puppet freshness on mw58 is CRITICAL: No successful Puppet run in the last 10 hours [15:41:49] PROBLEM - Puppet freshness on pdf1 is CRITICAL: No successful Puppet run in the last 10 hours [15:41:50] PROBLEM - Puppet freshness on sq42 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:26] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [16:08:56] PROBLEM - Puppet freshness on aluminium is CRITICAL: No successful Puppet run in the last 10 hours [16:08:56] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:56] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:56] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:56] PROBLEM - Puppet freshness on amssq33 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:56] PROBLEM - Puppet freshness on amssq31 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:56] PROBLEM - Puppet freshness on amssq35 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:55] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:55] PROBLEM - Puppet freshness on analytics1023 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:55] PROBLEM - Puppet freshness on db58 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:55] PROBLEM - Puppet freshness on db1029 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:55] PROBLEM - Puppet freshness on es6 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:55] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:55] PROBLEM - Puppet freshness on cp1036 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:55] PROBLEM - Puppet freshness on db1001 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:55] PROBLEM - Puppet freshness on db1031 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:55] PROBLEM - Puppet freshness on cp3009 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:55] PROBLEM - Puppet freshness on db34 is CRITICAL: No successful Puppet run in the last 10 hours [16:11:56] PROBLEM - Puppet freshness on cp1024 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:55] PROBLEM - Puppet freshness on cp1002 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:55] PROBLEM - Puppet freshness on cp1005 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:55] PROBLEM - Puppet freshness on cp1049 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:55] PROBLEM - Puppet freshness on db39 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:55] PROBLEM - Puppet freshness on ekrem is CRITICAL: No successful Puppet run in the last 10 hours [16:12:55] PROBLEM - Puppet freshness on helium is CRITICAL: No successful Puppet run in the last 10 hours [16:12:55] PROBLEM - Puppet freshness on mc1001 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:56] PROBLEM - Puppet freshness on ms10 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:56] PROBLEM - Puppet freshness on mw1043 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:57] PROBLEM - Puppet freshness on mw1063 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:57] PROBLEM - Puppet freshness on mw1087 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:58] PROBLEM - Puppet freshness on mw1106 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:58] PROBLEM - Puppet freshness on mw124 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:59] PROBLEM - Puppet freshness on mw20 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:59] PROBLEM - Puppet freshness on mw43 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:00] PROBLEM - Puppet freshness on mw57 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:00] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:01] PROBLEM - Puppet freshness on srv255 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:01] PROBLEM - Puppet freshness on stat1 is CRITICAL: No successful Puppet run in the last 10 hours [16:13:02] PROBLEM - Puppet freshness on titanium is CRITICAL: No successful Puppet run in the last 10 hours [16:52:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time [16:57:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:59:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [17:02:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:03:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [17:12:15] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [17:12:15] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:15] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:15] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:15] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:16] PROBLEM - Puppet freshness on sodium is CRITICAL: No successful Puppet run in the last 10 hours [17:12:16] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:16] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [17:12:16] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [17:13:15] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [17:16:05] is anybody with root around? [17:22:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:23:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time [17:52:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [18:08:47] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [18:10:55] New patchset: Yuvipanda; "Setup monitoring for Redis on toollabs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71181 [18:11:54] New review: coren; "We iz like monitoringz." [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/71181 [18:11:54] Change merged: coren; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71181 [18:22:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:23:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time [18:26:12] Has to be adatped to (a) start the backend on the grid and (b) track where on the grid it is running. [18:52:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:53:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.137 second response time [19:08:43] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [19:09:03] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [19:10:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:11:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [19:31:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [19:34:53] PROBLEM - Disk space on ms-be1001 is CRITICAL: DISK CRITICAL - free space: / 5689 MB (3% inode=98%): [19:49:54] New patchset: Odder; "(bug 50425) Add the flood flag on bs.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71243 [20:06:35] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [20:08:55] PROBLEM - Disk space on ms-be1002 is CRITICAL: DISK CRITICAL - free space: / 5689 MB (3% inode=98%): [20:16:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:17:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [20:32:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:33:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [20:42:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:43:22] New patchset: QChris; "Switch apache's reverse proxy config for gerrit back to ProxyPass" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71244 [20:43:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [20:52:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:53:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [21:07:45] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [21:22:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:24:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [21:33:44] Published patchset: QChris; "Replicate operations/puppet/cdh4 to puppet-cdh4 on github" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71248 [21:46:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:47:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [21:52:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:53:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [22:08:52] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [22:16:05] New patchset: Odder; "(bug 50287) Restrict local uploads on Meta-Wiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71252 [22:22:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:23:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.134 second response time [22:24:14] New patchset: Odder; "(bug 50287) Restrict local uploads on Meta-Wiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71252 [22:25:42] hmmmm [22:30:14] odder: do you run php -l *.php before uploading? it means you have a syntax error [22:30:38] legoktm: jenkins runs that, I see [22:30:59] yeah, but its also useful if you do it locally [22:31:07] same error. [22:31:20] well yeah [22:31:25] it means you have a syntax error somewhere [22:32:06] https://gerrit.wikimedia.org/r/#/c/71252/2/wmf-config/InitialiseSettings.php has an even number of () chars I think [22:36:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:37:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [22:43:41] New patchset: Odder; "(bug 50287) Restrict local uploads on Meta-Wiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/71252 [22:52:22] PROBLEM - Puppet freshness on wtp1015 is CRITICAL: No successful Puppet run in the last 10 hours [22:59:23] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [23:00:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:03:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time [23:06:22] PROBLEM - Puppet freshness on ms-be1001 is CRITICAL: No successful Puppet run in the last 10 hours [23:06:28] New patchset: Yuvipanda; "Add libraries required to run iPython Notebook" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/71253 [23:06:42] PROBLEM - NTP on ssl3003 is CRITICAL: NTP CRITICAL: No response from NTP server [23:09:46] New review: Yuvipanda; "WIP for a http://nbviewer.ipython.org/ style tool." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/71253 [23:10:03] PROBLEM - Puppet freshness on celsus is CRITICAL: No successful Puppet run in the last 10 hours [23:15:30] New review: Daniel Kinzler; "i'd be much in favor of an internal rewrite, but i'm told there are mysterious issues with doing that." [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/65443 [23:20:03] PROBLEM - NTP on ssl3002 is CRITICAL: NTP CRITICAL: No response from NTP server [23:22:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:24:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time [23:31:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:32:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time