[00:09:06] RECOVERY - SSH on ms1002 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [01:35:03] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [02:21:55] !log LocalisationUpdate completed (1.21wmf5) at Sun Dec 2 02:21:55 UTC 2012 [02:22:10] Logged the message, Master [02:43:34] !log LocalisationUpdate completed (1.21wmf4) at Sun Dec 2 02:43:34 UTC 2012 [02:43:41] Logged the message, Master [03:04:00] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [03:04:00] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [03:13:36] PROBLEM - LVS HTTPS IPv4 on wikipedia-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:13:36] PROBLEM - LVS HTTPS IPv6 on wikipedia-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:13:45] PROBLEM - LVS HTTP IPv6 on wikipedia-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:13:45] PROBLEM - LVS HTTPS IPv6 on foundation-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:14:03] PROBLEM - SSH on lvs1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:14:12] PROBLEM - LVS HTTP IPv6 on wikidata-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:14:12] PROBLEM - LVS HTTP IPv4 on wikiversity-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:14:30] PROBLEM - LVS HTTP IPv6 on wikiversity-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:14:39] PROBLEM - LVS HTTPS IPv6 on wikinews-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:14:39] PROBLEM - LVS HTTP IPv6 on wikisource-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:14:39] PROBLEM - LVS HTTP IPv6 on wikimedia-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:14:39] PROBLEM - LVS HTTP IPv4 on wikiquote-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:14:39] PROBLEM - LVS HTTPS IPv6 on wikibooks-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:14:48] PROBLEM - LVS HTTP IPv6 on wikibooks-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:14:48] PROBLEM - LVS HTTPS IPv6 on mediawiki-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:14:48] PROBLEM - LVS HTTP IPv6 on foundation-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:14:57] PROBLEM - LVS HTTPS IPv6 on wikimedia-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:14:57] PROBLEM - LVS HTTPS IPv6 on wikiversity-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:15:15] PROBLEM - LVS HTTPS IPv4 on bits-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:15:15] PROBLEM - LVS HTTPS IPv4 on wikimedia-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:15:16] PROBLEM - LVS HTTPS IPv6 on wikiquote-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:15:24] PROBLEM - LVS HTTPS IPv6 on wiktionary-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:15:24] PROBLEM - LVS HTTP IPv4 on mediawiki-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:15:42] PROBLEM - LVS HTTPS IPv6 on wikisource-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:16:09] RECOVERY - LVS HTTP IPv4 on wikiquote-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 60633 bytes in 3.675 seconds [03:16:27] RECOVERY - LVS HTTPS IPv6 on wikiversity-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.767 seconds [03:16:45] RECOVERY - LVS HTTPS IPv6 on wikipedia-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.178 seconds [03:16:45] RECOVERY - LVS HTTP IPv6 on wikipedia-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.145 seconds [03:16:46] RECOVERY - LVS HTTPS IPv6 on wikiquote-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.182 seconds [03:16:46] RECOVERY - LVS HTTPS IPv4 on wikimedia-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.1 200 OK - 89733 bytes in 0.261 seconds [03:16:46] RECOVERY - LVS HTTP IPv4 on mediawiki-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 60632 bytes in 0.154 seconds [03:16:46] RECOVERY - LVS HTTPS IPv6 on foundation-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.179 seconds [03:16:46] RECOVERY - LVS HTTPS IPv4 on wikipedia-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.1 200 OK - 89733 bytes in 0.241 seconds [03:16:47] RECOVERY - LVS HTTPS IPv6 on wiktionary-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.354 seconds [03:16:54] RECOVERY - LVS HTTPS IPv4 on bits-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.1 200 OK - 3840 bytes in 0.138 seconds [03:17:12] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [03:17:21] RECOVERY - LVS HTTP IPv4 on wikiversity-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 60632 bytes in 0.135 seconds [03:17:21] RECOVERY - LVS HTTP IPv6 on wikidata-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.164 second response time [03:17:21] RECOVERY - LVS HTTPS IPv6 on wikisource-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.167 seconds [03:17:30] RECOVERY - LVS HTTP IPv6 on wikiversity-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.135 seconds [03:17:39] RECOVERY - LVS HTTPS IPv6 on wikinews-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.168 seconds [03:17:39] RECOVERY - LVS HTTP IPv6 on wikimedia-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 89733 bytes in 0.145 seconds [03:17:39] RECOVERY - LVS HTTPS IPv6 on wikibooks-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.357 seconds [03:17:39] RECOVERY - LVS HTTP IPv6 on wikisource-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 1.138 seconds [03:17:48] RECOVERY - LVS HTTPS IPv6 on mediawiki-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.183 seconds [03:17:48] RECOVERY - LVS HTTP IPv6 on foundation-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.346 seconds [03:17:49] RECOVERY - LVS HTTP IPv6 on wikibooks-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.560 seconds [03:20:03] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [03:20:03] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [03:21:15] PROBLEM - LVS HTTP IPv4 on mobile-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:21:15] RECOVERY - LVS HTTPS IPv6 on wikimedia-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 89733 bytes in 0.837 seconds [03:21:42] PROBLEM - LVS HTTP IPv4 on wikipedia-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:21:42] PROBLEM - LVS HTTPS IPv4 on wikimedia-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:21:51] PROBLEM - LVS HTTPS IPv4 on wikipedia-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:22:18] PROBLEM - LVS HTTP IPv4 on wikiversity-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:22:36] PROBLEM - LVS HTTP IPv6 on wiktionary-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:22:36] PROBLEM - LVS HTTPS IPv6 on wikibooks-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:22:45] PROBLEM - LVS HTTPS IPv6 on wikinews-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:22:45] PROBLEM - LVS HTTP IPv6 on wikiversity-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:23:30] PROBLEM - LVS HTTPS IPv6 on foundation-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:23:33] just mailed ops [03:23:39] RECOVERY - LVS HTTP IPv4 on wikiversity-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 60635 bytes in 3.213 seconds [03:24:06] RECOVERY - LVS HTTPS IPv6 on wikibooks-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.349 seconds [03:24:33] PROBLEM - LVS HTTP IPv4 on wikidata-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:24:42] PROBLEM - LVS HTTPS IPv6 on wikiversity-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:24:51] PROBLEM - LVS HTTPS IPv6 on wikipedia-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway [03:24:53] PROBLEM - LVS HTTPS IPv6 on wikiquote-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway [03:25:00] jeremyb: about what? i just noticed meta is down but not sure if that's what the bots are complaining about since i have them on /ignore :( [03:25:03] PROBLEM - LVS HTTP IPv4 on wikimedia-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:25:12] PROBLEM - LVS HTTP IPv6 on wikipedia-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:25:17] PROBLEM - LVS HTTP IPv6 on wikiquote-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:25:42] ori-l: we've had 10+ mins of LVS flapping in here (nagios) and more than a few users complained in #-tech [03:25:52] PROBLEM - LVS HTTP IPv4 on wiktionary-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:25:56] ori-l: i also witnessed the issue firsthand and watchmouse is complaining too [03:25:57] PROBLEM - LVS HTTP IPv6 on mediawiki-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:26:06] RECOVERY - LVS HTTPS IPv6 on wikinews-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60254 bytes in 7.198 seconds [03:26:24] RECOVERY - LVS HTTP IPv4 on mobile-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.1 200 OK - 17855 bytes in 9.097 seconds [03:26:33] PROBLEM - LVS HTTPS IPv6 on wikimedia-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:27:36] RECOVERY - LVS HTTP IPv6 on wiktionary-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60252 bytes in 0.150 seconds [03:27:45] RECOVERY - LVS HTTP IPv6 on mediawiki-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60254 bytes in 8.555 seconds [03:27:45] PROBLEM - LVS HTTP IPv6 on wikimedia-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:27:54] RECOVERY - LVS HTTP IPv4 on wikidata-lb.eqiad.wikimedia.org is OK: HTTP OK - HTTP/1.0 301 Moved Permanently - 0.754 second response time [03:27:54] PROBLEM - LVS HTTP IPv6 on wikibooks-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:12] RECOVERY - LVS HTTPS IPv6 on wikiversity-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60254 bytes in 8.569 seconds [03:28:12] PROBLEM - LVS HTTPS IPv6 on wikidata-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:12] PROBLEM - LVS HTTPS IPv6 on bits-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:22] RECOVERY - LVS HTTP IPv4 on wikipedia-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 60633 bytes in 7.290 seconds [03:28:22] RECOVERY - LVS HTTP IPv4 on wikimedia-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 90119 bytes in 2.358 seconds [03:28:38] PROBLEM - LVS HTTPS IPv6 on wiktionary-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:38] RECOVERY - LVS HTTP IPv6 on wikiquote-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60254 bytes in 4.439 seconds [03:28:38] PROBLEM - Host wikiquote-lb.eqiad.wikimedia.org is DOWN: PING CRITICAL - Packet loss = 100% [03:28:39] PROBLEM - LVS HTTPS IPv4 on bits-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:28:40] PROBLEM - LVS HTTP IPv4 on foundation-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:29:41] PROBLEM - LVS HTTP IPv6 on wikinews-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:30:17] RECOVERY - LVS HTTP IPv6 on wikibooks-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60252 bytes in 3.205 seconds [03:30:35] RECOVERY - LVS HTTPS IPv6 on wikimedia-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 89734 bytes in 1.403 seconds [03:30:36] RECOVERY - LVS HTTPS IPv6 on wikidata-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 2.226 second response time [03:30:44] RECOVERY - LVS HTTP IPv4 on wiktionary-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 60631 bytes in 0.161 seconds [03:30:44] PROBLEM - LVS HTTP IPv6 on foundation-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:30:53] RECOVERY - LVS HTTPS IPv6 on bits-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 3851 bytes in 1.122 seconds [03:30:53] PROBLEM - LVS HTTP IPv4 on m.wikimedia.org is CRITICAL: Connection timed out [03:31:03] PROBLEM - LVS HTTPS IPv6 on wikisource-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:31:11] RECOVERY - LVS HTTPS IPv6 on wiktionary-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60254 bytes in 0.194 seconds [03:31:29] RECOVERY - LVS HTTP IPv4 on foundation-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 60631 bytes in 0.704 seconds [03:31:38] RECOVERY - LVS HTTP IPv6 on wikiversity-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60252 bytes in 6.816 seconds [03:31:48] PROBLEM - LVS HTTP IPv4 on wikibooks-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:32:41] PROBLEM - LVS HTTP IPv6 on wiktionary-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:32:41] PROBLEM - LVS HTTP IPv6 on wikiquote-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:32:51] RECOVERY - LVS HTTPS IPv6 on wikiquote-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 15973 bytes in 8.947 seconds [03:33:26] PROBLEM - LVS HTTP IPv6 on mediawiki-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:33:53] RECOVERY - LVS HTTP IPv4 on m.wikimedia.org is OK: HTTP OK HTTP/1.1 200 OK - 0.106 second response time [03:34:02] RECOVERY - Host wikiquote-lb.eqiad.wikimedia.org is UP: PING WARNING - Packet loss = 61%, RTA = 31.25 ms [03:34:11] PROBLEM - LVS HTTP IPv6 on wikisource-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:34:47] PROBLEM - LVS HTTP IPv4 on wikiversity-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:35:23] PROBLEM - LVS HTTP IPv6 on wikibooks-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:35:41] PROBLEM - LVS HTTPS IPv6 on wikimedia-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:35:41] PROBLEM - LVS HTTP IPv4 on wiktionary-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:35:41] PROBLEM - LVS HTTP IPv4 on wikipedia-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:35:50] RECOVERY - LVS HTTP IPv6 on wiktionary-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60255 bytes in 0.148 seconds [03:35:59] RECOVERY - LVS HTTPS IPv6 on foundation-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60255 bytes in 0.617 seconds [03:35:59] PROBLEM - LVS HTTP IPv4 on mediawiki-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:36:35] RECOVERY - LVS HTTP IPv6 on mediawiki-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 3.348 seconds [03:36:44] PROBLEM - LVS HTTP IPv6 on wikiversity-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:14] RECOVERY - LVS HTTP IPv4 on wikibooks-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 60632 bytes in 5.860 seconds [03:38:23] PROBLEM - LVS HTTP IPv4 on wikisource-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:32] RECOVERY - LVS HTTP IPv6 on wikibooks-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60255 bytes in 7.171 seconds [03:38:50] RECOVERY - LVS HTTP IPv6 on foundation-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.143 seconds [03:38:50] RECOVERY - LVS HTTP IPv4 on wiktionary-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 60632 bytes in 0.156 seconds [03:38:50] RECOVERY - LVS HTTP IPv6 on wikisource-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.157 seconds [03:38:50] RECOVERY - LVS HTTPS IPv6 on wikimedia-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 89734 bytes in 1.397 seconds [03:38:50] RECOVERY - LVS HTTP IPv4 on wikipedia-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 60632 bytes in 3.363 seconds [03:39:08] RECOVERY - LVS HTTP IPv4 on mediawiki-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 60632 bytes in 0.136 seconds [03:39:08] RECOVERY - LVS HTTP IPv6 on wikiquote-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.140 seconds [03:39:08] RECOVERY - LVS HTTPS IPv4 on wikipedia-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.1 200 OK - 89734 bytes in 0.204 seconds [03:39:08] RECOVERY - LVS HTTPS IPv6 on wikisource-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.177 seconds [03:39:35] RECOVERY - LVS HTTP IPv6 on wikipedia-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.136 seconds [03:39:35] RECOVERY - LVS HTTP IPv4 on wikiversity-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 60632 bytes in 0.135 seconds [03:39:35] RECOVERY - LVS HTTP IPv6 on wikinews-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.138 seconds [03:39:53] RECOVERY - LVS HTTP IPv6 on wikimedia-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 89734 bytes in 0.136 seconds [03:39:53] RECOVERY - LVS HTTP IPv6 on wikiversity-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60255 bytes in 0.136 seconds [03:39:53] RECOVERY - LVS HTTP IPv4 on wikisource-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 60634 bytes in 0.145 seconds [03:39:53] RECOVERY - LVS HTTPS IPv6 on wikipedia-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 60253 bytes in 0.169 seconds [03:41:40] hi [03:42:16] hey [03:42:27] hi [03:42:28] I'm not really able to do much from here [03:42:35] i'm not on my computer [03:44:17] lvs in traffic jumped quite a bit [03:45:21] yeah, something weird started in eqiad lvs around 3:03 UTC [03:45:56] well, I should go. I honestly won't be much help like this [03:46:05] k [03:46:10] ok [03:46:22] I should keep my key on a usb stick [03:46:22] pmtpa and esams also were weird around the same time but eqiad is more extreme [03:46:50] stuff is up now right? [03:48:06] watchmouse isn't happy yet but that's not conclusive [03:48:25] the page i've been trying is now loading ok but i should try a few others [03:49:54] also, surveyed a couple channels and no one's reporting problems right now [03:50:07] <^demon> Mobile seems to be fine too. [03:51:47] I've had anecdotal reports of banners misbehaving today. How's bits? [03:51:53] ...payments seems fine. [03:52:25] K4-713: before this hour? [03:52:36] Yeah. But I don't know how much I believe it. [04:09:17] PROBLEM - Puppet freshness on lvs4 is CRITICAL: Puppet has not run in the last 10 hours [04:09:17] PROBLEM - Puppet freshness on lvs3 is CRITICAL: Puppet has not run in the last 10 hours [04:22:11] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [05:25:11] New patchset: Catrope; "Fix typo that probably broke Parsoid LVS" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36376 [07:07:20] ori-l: you still online? [07:12:30] Change merged: preilly; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36376 [07:18:49] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [07:23:01] preilly: am now [07:23:51] you say pmtpa, i say ptmpta -- let's call the whole thing off [07:23:59] ori-l: ha ha [07:24:42] * ori-l deploys vowels to pmtpa [07:24:59] anyways, what's up? [07:25:30] ori-l: just curious if you looked at NavigationTiming at all? [07:25:46] * preilly knows he said don't worry about it [07:25:59] yeah, not ready for review though [07:26:16] i proooooomise by tomorrow evening [07:26:24] ori-l: ha ha [07:26:33] wasn't joking :) [07:26:34] ori-l: no worries… I was just curious [07:26:46] ori-l: I didn't think you were joking [07:28:08] i'm working on about 14 things at the moment :| [07:28:17] but the nav timing thing is exciting [07:28:36] ori-l: yeah [07:29:13] speaking of front-end performance, i want to either lobby for the outright removal of the sandbox link in the enwiki personal tools section or generate it in php [07:29:38] \o/ [07:29:52] there should be some convention that dom manipulation on DOMContentReady not alter the document flow [07:30:48] otherwise this annoying thing happens: page loads, you click 'Talk page', but the UI thread is busy; 'Sandbox' gets interpolated in, and your click ends up hitting an adjacent link [07:31:24] ori-l: yeah that needs to be fixed for sure [07:31:25] ori-l: errrrr, you mean what happened last year for the entire fundraiser? [07:31:42] what happened last year during the entire fundraiser? [07:31:55] page loads from a cache with 30 day ttl [07:32:07] so it can't have the space for a banner preallocated [07:32:23] because it won't necessarily be present for the whole life of the cached object [07:32:40] banner loads in ajax and then is added to DOM [07:33:07] click one of the links for tabs at the top of the page and you end up clicking the banner itself instead [07:33:28] x2 donations! \o/ [07:33:29] this year also has some serious issues though [07:33:31] j/k [07:33:36] that is annoying as hell, i agree [07:34:06] http://www.youtube.com/watch?v=1aBaX9GPSaQ [07:34:11] i wonder what's a good venue/forum/time to discuss the related problems with this year's most common banners (at least the ones i'm seeing) [07:34:19] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [07:35:21] ori-l: oh and have you seen https://bugzilla.wikimedia.org/26447 ? [07:35:44] dunno, maybe preilly knows. it's possible the fundraising team wants to just focus on keeping everything stable, in which case they ought to be indulged, IMHO, and UX issues should be flagged after the fundraiser concludes. but it is possible that they are interested in UX feedback. [07:36:09] i don't think so. i think this is worth fixing [07:36:18] sure, a/b test it and everything [07:36:21] but also fix it [07:36:23] ;) [07:36:43] It should be fixed [07:39:39] what fiscal year is this? 2012 or 2013 ? [07:40:21] 2012 [07:41:02] http://wikimediafoundation.org/wiki/Financial_reports#2012.E2.80.932013_fiscal_year [07:41:22] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours [07:42:46] preilly: ori-l: https://meta.wikimedia.org/wiki/Talk:Fundraising_2011/archive/2#Using_Javascript_to_display_banners [07:43:09] anyway, that style is apparently not in use any more? [08:10:20] jeremyb: we definitely want to know about UX issues [09:26:03] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [11:36:00] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [13:04:47] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [13:04:47] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [13:20:41] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [13:20:41] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [14:10:12] PROBLEM - Puppet freshness on lvs3 is CRITICAL: Puppet has not run in the last 10 hours [14:10:12] PROBLEM - Puppet freshness on lvs4 is CRITICAL: Puppet has not run in the last 10 hours [14:23:06] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [16:05:10] New review: Hashar; "When this change is merged, we need to update the ant build script properties ( see https://gerrit.w..." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/35159 [16:40:14] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:41:44] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 5.709 seconds [17:16:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:20:17] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [17:29:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.204 seconds [17:35:17] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [17:42:20] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours [18:04:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:19:05] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 9.020 seconds [18:54:03] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:11:36] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.021 seconds [19:27:12] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [19:43:42] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:58:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.032 seconds [20:31:11] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:47:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.033 seconds [21:21:08] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:37:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.025 seconds [21:37:20] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [22:05:52] New patchset: Alex Monk; "Revert "(bug 41358) Temporary celebration logo for ar.wikipedia"" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/36485 [22:09:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:16:33] tsk [22:16:45] Nemo_bis: ? [22:17:58] useless silly config changes [22:25:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.046 seconds [22:36:35] Nemo_bis: {{fact}} ??? [22:57:16] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:05:40] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [23:05:40] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [23:11:40] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 5.544 seconds [23:21:43] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [23:21:43] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [23:46:00] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds