[00:17:01] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [00:50:47] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [01:07:27] PROBLEM - RAID on analytics1010 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:09:27] RECOVERY - RAID on analytics1010 is OK: OK: Active: 6, Working: 6, Failed: 0, Spare: 0 [02:17:22] !log LocalisationUpdate completed (1.22wmf3) at Sun May 5 02:17:22 UTC 2013 [02:17:31] Logged the message, Master [02:25:45] !log LocalisationUpdate completed (1.22wmf2) at Sun May 5 02:25:44 UTC 2013 [02:25:52] Logged the message, Master [02:37:09] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [02:51:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:52:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [03:21:30] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:22:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [03:22:49] New patchset: Pgehres; "Removing myself from fundraising alerts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62318 [03:24:50] New patchset: Pgehres; "Removing myself from fundraising alerts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62319 [03:26:02] Change abandoned: Pgehres; "Duplicate of https://gerrit.wikimedia.org/r/#/c/62318/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62319 [03:36:17] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun May 5 03:36:17 UTC 2013 [03:36:26] Logged the message, Master [04:10:45] RECOVERY - Puppet freshness on mc15 is OK: puppet ran at Sun May 5 04:10:36 UTC 2013 [05:43:22] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [05:43:22] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [05:43:22] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [05:43:22] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [07:12:38] PROBLEM - Host mw1041 is DOWN: PING CRITICAL - Packet loss = 100% [07:39:23] New patchset: Faidon; "Varnish: fix mobile frontend's purge issues" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62324 [07:43:10] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62324 [10:17:56] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [10:48:26] jetty on vanadium is very slow, could someone check what is going on? [10:51:17] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [11:33:57] PROBLEM - Apache HTTP on mw1154 is CRITICAL: Connection timed out [11:34:17] PROBLEM - Apache HTTP on mw1158 is CRITICAL: Connection timed out [11:34:27] PROBLEM - LVS HTTP IPv4 on rendering.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:34:30] PROBLEM - Apache HTTP on mw1157 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:34:37] PROBLEM - Apache HTTP on mw1153 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:34:37] PROBLEM - Apache HTTP on mw1156 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:34:47] PROBLEM - Apache HTTP on mw1160 is CRITICAL: Connection timed out [11:34:47] PROBLEM - Apache HTTP on mw1159 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:34:47] PROBLEM - Apache HTTP on mw1155 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:37:37] RECOVERY - Apache HTTP on mw1155 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 2.681 second response time [11:37:38] RECOVERY - Apache HTTP on mw1159 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 4.214 second response time [11:40:17] RECOVERY - LVS HTTP IPv4 on rendering.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 64110 bytes in 7.491 second response time [11:40:19] RECOVERY - Apache HTTP on mw1157 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.048 second response time [11:40:27] RECOVERY - Apache HTTP on mw1153 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.059 second response time [11:40:27] RECOVERY - Apache HTTP on mw1156 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.052 second response time [11:40:37] RECOVERY - Apache HTTP on mw1160 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.048 second response time [11:41:17] RECOVERY - Apache HTTP on mw1158 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.045 second response time [11:41:27] RECOVERY - Apache HTTP on mw1154 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.047 second response time [11:47:37] PROBLEM - SSH on mc15 is CRITICAL: Connection timed out [11:48:27] RECOVERY - SSH on mc15 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:13:54] New patchset: QChris; "Turn on gerrit's database connection pooling" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62336 [12:15:13] New review: QChris; "Turning on connection pooling allows my local gerrit installation to" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62336 [12:37:55] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [13:21:26] Q: How many Wikimedia wikis are there? [13:56:38] odder: https://noc.wikimedia.org/conf/highlight.php?file=all.dblist [13:58:00] wc -l says 870 [13:58:49] 871 after pulling the current version ;) [14:05:58] Your account is active on 841 project sites. [14:06:01] Sounds about right [14:11:13] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [15:44:04] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [15:44:04] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [15:44:04] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [15:44:04] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [18:11:27] New patchset: Nikerabbit; "Enable input methods and web fonts for anonymous users" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62347 [18:12:49] New review: Nikerabbit; "For next Tuesday" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62347 [18:18:14] New review: Andrew Bogott; "Nobody cares about this but me!" [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/58922 [18:18:15] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/58922 [19:04:19] New patchset: Andrew Bogott; "Improvements to mediawiki_singlenode" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61816 [19:06:29] New review: Andrew Bogott; "I added back in that ::" [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/61816 [19:06:29] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61816 [20:18:20] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [21:04:22] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [21:13:22] New review: QChris; "I forgot to add that around 2009 there seem to have occurred some" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62336 [22:06:28] PROBLEM - HTTP on formey is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:06:38] PROBLEM - HTTPS on formey is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:23:38] RECOVERY - HTTPS on formey is OK: OK - Certificate will expire on 08/22/2015 22:23. [22:24:18] RECOVERY - HTTP on formey is OK: HTTP OK: HTTP/1.1 200 OK - 3596 bytes in 0.608 second response time [22:38:22] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [23:31:44] hey TimStarling, could I ask you to scp bugzilla's 'localconfig' (in the main bugzilla dir) and the apache site config from kaulen to somewhere readable by me? (if you want to scrub the file, localconfig has $db_pass and $site_wide_secret). i have access to fenari, stat1 & vanadium. [23:37:53] the apache site config is in puppet [23:41:09] huh, sorry. I don't know how I missed that. [23:42:32] localconfig is now in fenari:~olivneh [23:42:50] thanks.