[00:00:14] New patchset: Asher; "reassigning db1042 so s4 after db1020 hw failure" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62117 [00:01:05] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62117 [00:04:19] PROBLEM - mysqld processes on db1042 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [00:09:36] hashar: can i look at /var/log/apache2/integration_access.log ? [00:10:35] PROBLEM - DPKG on db1042 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [00:10:45] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [00:12:03] ori-l: would need /var/log/apache to be in the wikidev group [00:12:30] ori-l: oh you don't have access on gallium do you? I could gzip it on fenari if you want [00:12:49] yep, yep :) [00:15:35] RECOVERY - DPKG on db1042 is OK: All packages OK [00:15:39] !log DNS update - add iegcom [00:15:46] Logged the message, Master [00:20:05] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [00:22:00] New patchset: Dzahn; "add VirtualHost for new iegcom wiki, copied from other private wiki RT-5042" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62119 [00:24:29] Change merged: Aaron Schulz; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61425 [00:25:16] !log aaron synchronized multiversion/MWMultiVersion.php '6512451451ac6617bedbfce7be17045bed794d70' [00:25:24] Logged the message, Master [00:25:55] PROBLEM - Host db1042 is DOWN: PING CRITICAL - Packet loss = 100% [00:27:58] New patchset: Dzahn; "add document root for iegcom wiki, copied from skel-1.5 (RT-5042)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62120 [00:28:16] RECOVERY - Host db1042 is UP: PING OK - Packet loss = 0%, RTA = 0.95 ms [00:29:06] New review: Dzahn; "copied from skel1.5 per docs" [operations/mediawiki-config] (master) C: 2; - https://gerrit.wikimedia.org/r/62120 [00:29:12] New review: Dzahn; "copied from skel1.5 per docs" [operations/mediawiki-config] (master); V: 2 - https://gerrit.wikimedia.org/r/62120 [00:29:12] Change merged: Dzahn; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62120 [00:31:03] Change merged: Aaron Schulz; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61426 [00:32:01] !log sync-docroot for iegcom wiki [00:32:09] Logged the message, Master [00:32:22] !log dzahn synchronized docroot [00:32:30] Logged the message, Master [00:39:49] hashar: do we still need that secure.wikimedia.org code in multiversion? [00:40:12] aren't we keeping secure for back compatibility ? [00:40:39] Aaron|home: no [00:41:04] Aaron|home, hashar: secure.wm.org is just a bunch of apache redirects now [00:41:12] wonderful! [00:41:24] that's what I thought [00:42:05] mutante: let's deploy that change btw [00:48:50] New patchset: Krinkle; "contint: Add rewrite rules for favicon.ico to favicon.php" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62125 [00:50:50] hashar: mutante: https://integration.wikimedia.org/favicon.php https://gerrit.wikimedia.org/r/62125 [00:51:09] New review: Krinkle; "obsolete with Id483de4f8963840e" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/62108 [00:51:31] php? why? [00:51:44] paravoid: same as production [00:51:52] paravoid: all our favicons are served through php [00:52:05] https://en.wikipedia.org/w/favicon.php [00:56:05] paravoid: secure? hrmm i couldn't immediately confirm it working on a random test host and tomorrow at ulsfo, but .. yeah! [00:57:10] re: favicon, well, if it's like on the wikis ... [00:57:16] i gotta go though.. meeting [01:01:48] New review: Tim Starling; "Maybe Robert imagined that MW would be smart enough to not send empty queries, and the error is ther..." [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/60860 [01:05:59] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [01:24:58] !log Zuul processed all its events, seems to be fine now. *sleep* [01:25:03] have a good night everyone [01:25:06] Logged the message, Master [01:32:40] good night hashar [01:32:47] thx :-] [02:06:13] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [02:15:45] !log LocalisationUpdate completed (1.22wmf3) at Fri May 3 02:15:45 UTC 2013 [02:15:53] Logged the message, Master [02:26:03] !log LocalisationUpdate completed (1.22wmf2) at Fri May 3 02:26:02 UTC 2013 [02:26:11] Logged the message, Master [03:06:01] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [03:40:15] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [03:40:15] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [03:40:15] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [03:40:15] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [04:03:13] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri May 3 04:03:12 UTC 2013 [04:03:23] Logged the message, Master [04:04:23] New patchset: Jeremyb; "reenable rfaulkner round 3" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62131 [04:04:29] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:05:30] New review: Jeremyb; "left a question in RT." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/62131 [04:08:19] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 04:08:15 UTC 2013 [04:08:29] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:09] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 04:10:04 UTC 2013 [04:10:29] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:11:49] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 04:11:43 UTC 2013 [04:12:29] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:13:29] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 04:13:22 UTC 2013 [04:14:29] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:59] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 04:14:52 UTC 2013 [04:15:29] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:16:21] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 04:16:14 UTC 2013 [04:16:29] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:17:39] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 04:17:33 UTC 2013 [04:18:29] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:18:49] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 04:18:43 UTC 2013 [04:19:29] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:19:49] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [04:19:49] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 04:19:48 UTC 2013 [04:20:29] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:20:49] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 04:20:46 UTC 2013 [04:21:29] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:21:39] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 04:21:36 UTC 2013 [04:22:29] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:22:59] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 04:22:57 UTC 2013 [04:23:29] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:23:39] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 04:23:32 UTC 2013 [04:24:29] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [04:25:45] New patchset: Ori.livneh; "Allow multiple CIDR ranges to be specified" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61920 [04:25:46] New patchset: Ori.livneh; "logmsgbot: only allow connections from tin, fenari & localhost" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61931 [04:26:12] New review: Ori.livneh; "PS2 allows multiple values to be specified as a Puppet array, per hashar's request." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61920 [04:26:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:27:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.153 second response time [04:33:49] RECOVERY - search indices - check lucene status page on search18 is OK: HTTP OK: HTTP/1.1 200 OK - 55880 bytes in 0.116 second response time [04:57:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:59:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [05:07:59] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [05:47:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:48:26] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [06:06:21] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [06:15:56] !log tstarling synchronized wmf-config/missing.php [06:16:05] Logged the message, Master [06:19:51] New patchset: Tim Starling; "Fix use of PATH_INFO in missing.php" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62136 [06:20:32] Change merged: Tim Starling; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62136 [06:30:11] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 06:30:06 UTC 2013 [06:30:21] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [06:31:01] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 06:30:51 UTC 2013 [06:31:21] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [06:31:41] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 06:31:31 UTC 2013 [06:31:43] New patchset: Tim Starling; "Allow multiple CIDR ranges to be specified" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61920 [06:31:51] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61920 [06:32:17] New patchset: Tim Starling; "logmsgbot: only allow connections from tin, fenari & localhost" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61931 [06:32:21] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [06:32:25] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61931 [06:40:18] notice: Skipping run of Puppet configuration client; administratively disabled; use 'puppet Puppet configuration client --enable' to re-enable. [06:40:21] what is this? [06:42:41] a very unhelpful error message emitted whenever the puppet client detects puppet agent isn't running [06:43:08] googling it is funny because people seem uniformly surprised by it [06:44:51] it looks like it's running to me [06:45:02] well, after I started it, but still the same message [06:45:32] it's not meant to be running is it? [06:45:32] run 'puppet agent --enable'? [06:46:51] why would it be disabled? [06:46:52] it is in general, but someone may have disabled it on purpose. i had to resort to that once to prevent puppet from clobbering a live-hack i need to stay in place for longer than half an hour. [06:47:28] it just ran a few minutes ago according to the timestamp on puppet.log [06:47:30] i asked here and the speed with which i got an answer suggests the practice was not entirely uncommon [06:47:45] in fact, the first time I ran it, it said a puppet run was in progress, I had to wait a couple of minutes [06:47:48] so maybe it disabled itself? [06:48:06] hmm, would hitting the timeout do this? [06:48:36] the manual section for puppet agent --enable says that it removes a lock file [06:49:04] and it does take a huge amount of time on neon, because of icinga [06:49:33] dunno [06:58:01] well, it's been running for more than 10 minutes now [06:58:20] the timeout in the cron job is 5 [06:58:47] what is it doing? [06:59:07] sorry, timeout -k 300 1800, this is actually a 30 minute timeout [06:59:15] plus kill -9 at the 35 minute mark [06:59:27] presumably it is waiting for its catalogue [06:59:40] which will be enormous due to monitoring entries [06:59:42] gtg [07:00:16] Tim-pumpkin [07:02:42] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [07:51:13] hello, MaxSem [07:51:45] !log OMG logmsgbot is a terminator! [07:51:53] Logged the message, Master [07:56:08] lol [08:06:33] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [08:07:43] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 08:07:40 UTC 2013 [08:08:33] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:03] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 08:08:54 UTC 2013 [08:09:33] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [08:13:03] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 08:12:55 UTC 2013 [08:13:33] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [08:15:13] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:12] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [08:49:12] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [08:58:42] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:00:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [09:06:32] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [09:41:52] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [10:06:33] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [10:21:03] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [10:22:23] PROBLEM - Apache HTTP on mw1153 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:22:43] PROBLEM - Apache HTTP on mw1160 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:03] PROBLEM - Apache HTTP on mw1159 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:23] PROBLEM - Apache HTTP on mw1156 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:43] PROBLEM - LVS HTTP IPv4 on rendering.svc.eqiad.wmnet is CRITICAL: Connection timed out [10:23:44] PROBLEM - Apache HTTP on mw1155 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:23:44] PROBLEM - Apache HTTP on mw1158 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:24:23] PROBLEM - Apache HTTP on mw1157 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:24:33] PROBLEM - Apache HTTP on mw1154 is CRITICAL: Connection timed out [10:26:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:27:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.370 second response time [10:30:33] RECOVERY - Apache HTTP on mw1160 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.048 second response time [10:30:33] RECOVERY - Apache HTTP on mw1155 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.055 second response time [10:30:33] RECOVERY - LVS HTTP IPv4 on rendering.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 64442 bytes in 0.146 second response time [10:30:43] RECOVERY - Apache HTTP on mw1158 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 6.910 second response time [10:31:03] RECOVERY - Apache HTTP on mw1159 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 9.962 second response time [10:31:13] RECOVERY - Apache HTTP on mw1156 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.043 second response time [10:31:13] RECOVERY - Apache HTTP on mw1153 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 2.452 second response time [10:32:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:33:13] RECOVERY - Apache HTTP on mw1157 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.061 second response time [10:33:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time [10:33:53] New review: Burthsceh; "Because single quotations are used more than double quotations, I replaced quotes." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61782 [10:34:31] RECOVERY - Apache HTTP on mw1154 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.074 second response time [10:57:37] New patchset: Burthsceh; "(bug 47933) Rename Module talk namespace for Japanese Wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/61782 [11:01:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:02:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [11:05:55] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [11:26:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:29:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [12:01:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:25] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [12:06:17] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [12:07:57] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 12:07:47 UTC 2013 [12:08:17] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [12:09:07] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 12:08:58 UTC 2013 [12:09:17] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [12:10:07] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 12:10:00 UTC 2013 [12:10:17] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:07] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 12:10:57 UTC 2013 [12:11:17] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [12:11:57] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 12:11:47 UTC 2013 [12:12:17] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:37] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 12:12:30 UTC 2013 [12:13:17] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [12:13:47] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 12:13:37 UTC 2013 [12:14:01] !log Connected and brought up cr1-esams:ae1 <--> csw2-esams:ae1 [12:14:17] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [12:25:35] !log Disabled csw1-esams:ve5 (Backup/storage subnet), activated cr1-esams:ae1.101, VRRP for this subnet transitioned to cr1-esams:ae1.101 [12:28:37] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:30:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [12:32:50] New patchset: Matmarex; "(bug 48043) Activate Flagged Revisions for modules namespace on Polish Wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62147 [12:43:52] ~. [12:46:27] Password: [12:47:13] GrkBnkrptcy$000 [12:48:51] hahaha [12:48:54] touche [12:49:04] !log Disabled OSPF/OSPFv3 on cr2-knams:xe-0/0/0 <--> csw1-esams:e8/2 (GBLX DF leg 1) [12:51:25] !log Disabled csw1-esams:e8/2 [12:53:27] PROBLEM - Backend Squid HTTP on knsq28 is CRITICAL: Connection timed out [12:53:27] PROBLEM - HTTPS on ssl3001 is CRITICAL: Connection timed out [12:53:37] PROBLEM - Backend Squid HTTP on knsq27 is CRITICAL: Connection timed out [12:53:37] PROBLEM - LVS HTTPS IPv4 on mediawiki-lb.esams.wikimedia.org is CRITICAL: Connection timed out [12:53:37] PROBLEM - LVS HTTP IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 7.179 second response time [12:53:37] PROBLEM - LVS HTTP IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 7.181 second response time [12:53:37] PROBLEM - LVS HTTP IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 7.186 second response time [12:53:37] PROBLEM - LVS HTTP IPv6 on wikinews-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 7.190 second response time [12:53:37] PROBLEM - LVS HTTP IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 7.191 second response time [12:53:39] PROBLEM - LVS HTTPS IPv6 on foundation-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:39] PROBLEM - LVS HTTPS IPv4 on bits.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:39] PROBLEM - SSH on ssl3002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:39] PROBLEM - Backend Squid HTTP on knsq29 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:40] PROBLEM - SSH on knsq28 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:40] PROBLEM - HTTPS on ssl3003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:41] PROBLEM - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:41] PROBLEM - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:42] sigh [12:53:48] PROBLEM - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:48] PROBLEM - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:48] PROBLEM - LVS HTTPS IPv6 on bits-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:48] PROBLEM - LVS HTTPS IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:48] PROBLEM - LVS HTTPS IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:48] PROBLEM - LVS HTTPS IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:48] PROBLEM - LVS HTTP IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:48] PROBLEM - SSH on knsq26 is CRITICAL: Connection timed out [12:53:48] PROBLEM - Backend Squid HTTP on knsq23 is CRITICAL: Connection timed out [12:53:48] PROBLEM - LVS HTTP IPv6 on wikipedia-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:57] PROBLEM - LVS HTTPS IPv6 on wikimedia-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:57] PROBLEM - LVS HTTP IPv6 on foundation-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:57] PROBLEM - Frontend Squid HTTP on knsq23 is CRITICAL: Connection timed out [12:53:57] PROBLEM - LVS HTTPS IPv6 on upload-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:53:58] PROBLEM - SSH on knsq27 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:54:27] RECOVERY - Backend Squid HTTP on knsq28 is OK: HTTP OK: HTTP/1.0 200 OK - 1421 bytes in 0.177 second response time [12:54:27] RECOVERY - HTTPS on ssl3001 is OK: OK - Certificate will expire on 01/20/2016 12:00. [12:54:27] RECOVERY - LVS HTTPS IPv6 on foundation-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64911 bytes in 0.698 second response time [12:54:27] RECOVERY - Backend Squid HTTP on knsq29 is OK: HTTP OK: HTTP/1.0 200 OK - 1421 bytes in 0.175 second response time [12:54:27] RECOVERY - Backend Squid HTTP on knsq27 is OK: HTTP OK: HTTP/1.0 200 OK - 1420 bytes in 0.178 second response time [12:54:27] RECOVERY - SSH on ssl3002 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [12:54:27] RECOVERY - SSH on knsq28 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [12:54:28] RECOVERY - LVS HTTP IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64914 bytes in 0.346 second response time [12:54:28] RECOVERY - LVS HTTP IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64914 bytes in 0.350 second response time [12:54:29] RECOVERY - LVS HTTP IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64911 bytes in 0.353 second response time [12:54:29] RECOVERY - LVS HTTP IPv6 on wikinews-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64911 bytes in 0.349 second response time [12:54:30] RECOVERY - LVS HTTP IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64911 bytes in 0.351 second response time [12:54:30] RECOVERY - HTTPS on ssl3003 is OK: OK - Certificate will expire on 01/20/2016 12:00. [12:54:31] RECOVERY - LVS HTTPS IPv4 on bits.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 3917 bytes in 0.444 second response time [12:54:31] RECOVERY - LVS HTTPS IPv4 on mediawiki-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1030 bytes in 0.448 second response time [12:54:32] RECOVERY - LVS HTTPS IPv6 on bits-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 3914 bytes in 0.444 second response time [12:54:32] RECOVERY - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 49820 bytes in 0.611 second response time [12:54:33] RECOVERY - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 47332 bytes in 0.612 second response time [12:54:33] RECOVERY - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 40860 bytes in 0.622 second response time [12:54:37] RECOVERY - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 53738 bytes in 0.632 second response time [12:54:37] RECOVERY - LVS HTTPS IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64914 bytes in 0.705 second response time [12:54:37] RECOVERY - LVS HTTPS IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64914 bytes in 0.715 second response time [12:54:37] RECOVERY - LVS HTTP IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64914 bytes in 0.440 second response time [12:54:37] RECOVERY - LVS HTTPS IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64914 bytes in 0.722 second response time [12:54:37] RECOVERY - SSH on knsq26 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [12:54:37] RECOVERY - Backend Squid HTTP on knsq23 is OK: HTTP OK: HTTP/1.0 200 OK - 1421 bytes in 0.172 second response time [12:54:37] RECOVERY - LVS HTTP IPv6 on wikipedia-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64914 bytes in 0.435 second response time [12:54:47] RECOVERY - LVS HTTP IPv6 on foundation-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64914 bytes in 0.436 second response time [12:54:47] RECOVERY - LVS HTTPS IPv6 on wikimedia-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 89578 bytes in 0.712 second response time [12:54:47] RECOVERY - LVS HTTPS IPv6 on upload-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 562 bytes in 0.443 second response time [12:54:50] RECOVERY - Frontend Squid HTTP on knsq23 is OK: HTTP OK: HTTP/1.0 200 OK - 1415 bytes in 0.172 second response time [12:54:50] RECOVERY - SSH on knsq27 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [12:55:09] i hate foundry [12:55:22] !log Reenabled csw1-esams:e8/2 [13:00:32] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61990 [13:07:05] !log Disabled PyBal on amslvs2 [13:07:21] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [13:12:22] !log Migrated PyBal BGP peerings of amslvs1/2 from csw1-esams to cr1-esams [13:14:44] !log Disabled csw1-esams:e8/2 [13:16:51] PROBLEM - LVS HTTPS IPv4 on mediawiki-lb.esams.wikimedia.org is CRITICAL: Connection timed out [13:17:01] PROBLEM - HTTPS on ssl3002 is CRITICAL: Connection timed out [13:17:01] PROBLEM - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: Connection timed out [13:17:01] PROBLEM - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is CRITICAL: Connection timed out [13:17:01] PROBLEM - LVS HTTPS IPv6 on upload-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:03] PROBLEM - LVS HTTP IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 7.190 second response time [13:17:03] PROBLEM - LVS HTTP IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 7.190 second response time [13:17:03] PROBLEM - LVS HTTP IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 7.186 second response time [13:17:03] PROBLEM - LVS HTTPS IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:03] PROBLEM - LVS HTTPS IPv6 on wikinews-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:11] PROBLEM - Frontend Squid HTTP on knsq27 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:11] PROBLEM - LVS HTTPS IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:11] PROBLEM - LVS HTTPS IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:11] PROBLEM - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:11] PROBLEM - LVS HTTP IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:12] PROBLEM - LVS HTTP IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:12] PROBLEM - LVS HTTPS IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:21] PROBLEM - LVS HTTPS IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:21] PROBLEM - SSH on knsq24 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:21] PROBLEM - Backend Squid HTTP on knsq24 is CRITICAL: Connection timed out [13:17:21] PROBLEM - LVS HTTP IPv6 on wikinews-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 7.183 second response time [13:17:21] PROBLEM - LVS HTTPS IPv4 on wikipedia-lb.esams.wikimedia.org is CRITICAL: Connection timed out [13:17:21] PROBLEM - SSH on ssl3003 is CRITICAL: Connection timed out [13:17:21] PROBLEM - Frontend Squid HTTP on knsq24 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:22] PROBLEM - LVS HTTPS IPv4 on upload.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:22] PROBLEM - LVS HTTPS IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:23] PROBLEM - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:23] PROBLEM - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:24] PROBLEM - LVS HTTP IPv6 on wikimedia-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:25] PROBLEM - Backend Squid HTTP on knsq28 is CRITICAL: Connection timed out [13:17:31] PROBLEM - LVS HTTPS IPv4 on bits.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:31] PROBLEM - HTTPS on ssl3003 is CRITICAL: Connection timed out [13:17:31] PROBLEM - LVS HTTP IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 325 bytes in 7.185 second response time [13:17:41] PROBLEM - Frontend Squid HTTP on knsq26 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:41] PROBLEM - Backend Squid HTTP on knsq23 is CRITICAL: Connection timed out [13:17:41] PROBLEM - HTTPS on ssl3001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:41] PROBLEM - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is CRITICAL: Connection timed out [13:17:41] PROBLEM - LVS HTTP IPv6 on wikipedia-lb.esams.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:51] RECOVERY - LVS HTTPS IPv4 on mediawiki-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1030 bytes in 0.462 second response time [13:17:51] RECOVERY - LVS HTTPS IPv6 on upload-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 559 bytes in 0.446 second response time [13:17:53] RECOVERY - HTTPS on ssl3002 is OK: OK - Certificate will expire on 01/20/2016 12:00. [13:17:53] RECOVERY - LVS HTTP IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64860 bytes in 0.437 second response time [13:17:53] RECOVERY - LVS HTTP IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64860 bytes in 0.440 second response time [13:17:53] RECOVERY - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 47336 bytes in 0.612 second response time [13:17:53] RECOVERY - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 60404 bytes in 0.726 second response time [13:17:53] RECOVERY - LVS HTTPS IPv6 on wikinews-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64857 bytes in 0.708 second response time [13:17:53] RECOVERY - LVS HTTPS IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64860 bytes in 0.716 second response time [13:18:02] RECOVERY - Frontend Squid HTTP on knsq27 is OK: HTTP OK: HTTP/1.0 200 OK - 1415 bytes in 0.178 second response time [13:18:02] RECOVERY - LVS HTTP IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64857 bytes in 0.392 second response time [13:18:02] RECOVERY - LVS HTTP IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64860 bytes in 0.350 second response time [13:18:02] RECOVERY - LVS HTTP IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64860 bytes in 0.350 second response time [13:18:02] RECOVERY - LVS HTTPS IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64860 bytes in 0.703 second response time [13:18:02] RECOVERY - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 74081 bytes in 0.706 second response time [13:18:02] RECOVERY - LVS HTTPS IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64860 bytes in 0.710 second response time [13:18:02] RECOVERY - LVS HTTPS IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64860 bytes in 0.700 second response time [13:18:11] RECOVERY - SSH on knsq24 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:18:11] RECOVERY - LVS HTTPS IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64857 bytes in 0.711 second response time [13:18:11] RECOVERY - Frontend Squid HTTP on knsq24 is OK: HTTP OK: HTTP/1.0 200 OK - 1413 bytes in 0.175 second response time [13:18:11] RECOVERY - Backend Squid HTTP on knsq24 is OK: HTTP OK: HTTP/1.0 200 OK - 1420 bytes in 0.174 second response time [13:18:11] RECOVERY - LVS HTTP IPv6 on wikinews-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64860 bytes in 0.345 second response time [13:18:11] RECOVERY - LVS HTTPS IPv4 on upload.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 563 bytes in 0.444 second response time [13:18:11] RECOVERY - LVS HTTPS IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64857 bytes in 0.702 second response time [13:18:12] RECOVERY - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 53741 bytes in 0.698 second response time [13:18:12] RECOVERY - SSH on ssl3003 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [13:18:13] RECOVERY - LVS HTTP IPv6 on wikimedia-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 89577 bytes in 0.516 second response time [13:18:13] RECOVERY - LVS HTTPS IPv4 on wikipedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 64860 bytes in 0.719 second response time [13:18:14] RECOVERY - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 89576 bytes in 0.811 second response time [13:18:21] RECOVERY - LVS HTTPS IPv4 on bits.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 3904 bytes in 0.444 second response time [13:18:21] RECOVERY - Backend Squid HTTP on knsq28 is OK: HTTP OK: HTTP/1.0 200 OK - 1420 bytes in 0.175 second response time [13:18:21] RECOVERY - HTTPS on ssl3003 is OK: OK - Certificate will expire on 01/20/2016 12:00. [13:18:21] RECOVERY - LVS HTTP IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64860 bytes in 0.386 second response time [13:18:31] RECOVERY - Frontend Squid HTTP on knsq26 is OK: HTTP OK: HTTP/1.0 200 OK - 1415 bytes in 0.175 second response time [13:18:31] RECOVERY - HTTPS on ssl3001 is OK: OK - Certificate will expire on 01/20/2016 12:00. [13:18:31] RECOVERY - Backend Squid HTTP on knsq23 is OK: HTTP OK: HTTP/1.0 200 OK - 1420 bytes in 0.409 second response time [13:18:31] RECOVERY - LVS HTTP IPv6 on wikipedia-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64857 bytes in 0.436 second response time [13:18:41] RECOVERY - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 40859 bytes in 0.612 second response time [13:35:44] ok [13:35:46] then the butcher method [13:37:27] !log Activated cr1-esams:ae1.100 VRRP (as master), disabled csw1-esams:ve1 [13:38:44] PROBLEM - LVS HTTP IPv6 on upload-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:38:49] PROBLEM - LVS HTTP IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:38:55] morebots :/ [13:39:04] PROBLEM - LVS HTTP IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:39:14] PROBLEM - LVS HTTP IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:39:14] PROBLEM - LVS HTTPS IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:39:14] PROBLEM - LVS HTTPS IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:39:14] PROBLEM - LVS HTTP IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:39:14] PROBLEM - LVS HTTPS IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:39:14] PROBLEM - LVS HTTPS IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:39:24] PROBLEM - LVS HTTPS IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:39:24] PROBLEM - LVS HTTPS IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:39:24] PROBLEM - LVS HTTP IPv6 on wikinews-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:39:24] PROBLEM - LVS HTTP IPv6 on wikimedia-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:39:24] PROBLEM - LVS HTTP IPv6 on foundation-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:39:24] PROBLEM - LVS HTTPS IPv6 on wikinews-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:39:34] PROBLEM - LVS HTTP IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:39:34] PROBLEM - LVS HTTPS IPv6 on foundation-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:39:34] PROBLEM - LVS HTTP IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:39:44] PROBLEM - LVS HTTP IPv6 on wikipedia-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:39:46] PROBLEM - LVS HTTPS IPv6 on bits-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:39:47] PROBLEM - LVS HTTPS IPv6 on wikimedia-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:39:47] PROBLEM - LVS HTTPS IPv6 on wikipedia-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:40:04] PROBLEM - LVS HTTPS IPv6 on upload-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out [13:41:04] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [13:41:04] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [13:41:04] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [13:41:04] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [13:44:34] PROBLEM - LVS HTTP IPv4 on foundation-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:44:54] PROBLEM - LVS HTTPS IPv4 on wiktionary-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:44:54] PROBLEM - LVS HTTP IPv4 on wikibooks-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:04] PROBLEM - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:14] PROBLEM - LVS HTTP IPv4 on wikiversity-lb.esams.wikimedia.org is CRITICAL: Connection timed out [13:45:14] PROBLEM - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:24] PROBLEM - LVS HTTP IPv4 on wikisource-lb.esams.wikimedia.org is CRITICAL: Connection timed out [13:45:24] PROBLEM - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:24] PROBLEM - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:24] PROBLEM - LVS HTTP IPv4 on wiktionary-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:24] PROBLEM - Frontend Squid HTTP on amssq55 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:54] PROBLEM - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:14] RECOVERY - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 73887 bytes in 8.425 second response time [13:46:24] PROBLEM - LVS HTTP IPv4 on wikinews-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:24] PROBLEM - SSH on amssq42 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:24] PROBLEM - LVS HTTP IPv4 on wikimedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:25] PROBLEM - LVS HTTPS IPv4 on wikipedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:34] PROBLEM - SSH on amssq50 is CRITICAL: Connection timed out [13:46:34] PROBLEM - SSH on amssq45 is CRITICAL: Connection timed out [13:46:54] RECOVERY - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 60401 bytes in 3.207 second response time [13:47:04] PROBLEM - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:14] PROBLEM - SSH on amssq47 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:24] RECOVERY - LVS HTTP IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 49531 bytes in 5.039 second response time [13:47:24] RECOVERY - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 89391 bytes in 8.400 second response time [13:47:34] RECOVERY - SSH on amssq50 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:47:34] RECOVERY - SSH on amssq45 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:47:44] PROBLEM - SSH on amssq55 is CRITICAL: Connection timed out [13:47:44] RECOVERY - LVS HTTP IPv4 on wikibooks-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 46863 bytes in 0.348 second response time [13:47:44] PROBLEM - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:54] PROBLEM - Backend Squid HTTP on amssq35 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:54] PROBLEM - LVS HTTP IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:48:04] RECOVERY - SSH on amssq47 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:48:14] PROBLEM - Frontend Squid HTTP on amssq32 is CRITICAL: Connection timed out [13:48:24] RECOVERY - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 53738 bytes in 7.208 second response time [13:48:24] RECOVERY - Frontend Squid HTTP on amssq55 is OK: HTTP OK: HTTP/1.0 200 OK - 565 bytes in 8.226 second response time [13:48:44] RECOVERY - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 40857 bytes in 0.822 second response time [13:48:44] PROBLEM - Frontend Squid HTTP on amssq59 is CRITICAL: Connection timed out [13:48:44] RECOVERY - LVS HTTP IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 59931 bytes in 1.534 second response time [13:48:44] RECOVERY - Backend Squid HTTP on amssq35 is OK: HTTP OK: HTTP/1.0 200 OK - 1422 bytes in 2.020 second response time [13:48:54] RECOVERY - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 49818 bytes in 8.335 second response time [13:49:04] RECOVERY - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 47104 bytes in 8.317 second response time [13:49:34] RECOVERY - SSH on amssq55 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:49:44] RECOVERY - Frontend Squid HTTP on amssq59 is OK: HTTP OK: HTTP/1.0 200 OK - 567 bytes in 1.434 second response time [13:50:14] RECOVERY - SSH on amssq42 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:50:44] PROBLEM - SSH on amssq31 is CRITICAL: Connection timed out [13:50:44] RECOVERY - LVS HTTPS IPv4 on wiktionary-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 69696 bytes in 0.710 second response time [13:51:14] PROBLEM - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:51:24] PROBLEM - Frontend Squid HTTP on amssq53 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:51:24] PROBLEM - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:51:24] PROBLEM - SSH on amssq52 is CRITICAL: Connection timed out [13:51:24] PROBLEM - SSH on amssq39 is CRITICAL: Connection timed out [13:51:44] PROBLEM - Backend Squid HTTP on amssq51 is CRITICAL: Connection timed out [13:51:44] PROBLEM - SSH on amssq61 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:51:54] PROBLEM - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:51:54] PROBLEM - LVS HTTP IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:52:14] RECOVERY - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 73887 bytes in 8.440 second response time [13:52:14] PROBLEM - SSH on amssq60 is CRITICAL: Connection timed out [13:52:14] RECOVERY - Frontend Squid HTTP on amssq53 is OK: HTTP OK: HTTP/1.0 200 OK - 567 bytes in 1.459 second response time [13:52:14] RECOVERY - LVS HTTPS IPv4 on wikipedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 64857 bytes in 1.719 second response time [13:52:14] RECOVERY - SSH on amssq52 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:52:24] RECOVERY - SSH on amssq39 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:52:24] RECOVERY - LVS HTTP IPv4 on wikimedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 89114 bytes in 9.861 second response time [13:52:33] New review: Andrew Bogott; "There's not a straightforward way to have per-directory .pep8 files; despite what pep8's usage strin..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61999 [13:52:34] RECOVERY - SSH on amssq61 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:52:44] RECOVERY - Backend Squid HTTP on amssq51 is OK: HTTP OK: HTTP/1.0 200 OK - 663 bytes in 2.021 second response time [13:52:44] PROBLEM - SSH on amssq35 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:52:54] RECOVERY - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 49820 bytes in 8.341 second response time [13:53:04] RECOVERY - SSH on amssq60 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:53:14] PROBLEM - Frontend Squid HTTP on amssq50 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:24] RECOVERY - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 89577 bytes in 5.451 second response time [13:53:34] PROBLEM - LVS HTTP IPv4 on wikipedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:53:44] RECOVERY - SSH on amssq31 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:53:44] RECOVERY - SSH on amssq35 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [13:53:44] PROBLEM - Frontend Squid HTTP on amssq41 is CRITICAL: Connection timed out [13:53:54] RECOVERY - LVS HTTP IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 59927 bytes in 5.071 second response time [13:54:09] PROBLEM - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:09] PROBLEM - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:14] RECOVERY - Frontend Squid HTTP on amssq50 is OK: HTTP OK: HTTP/1.0 200 OK - 565 bytes in 6.744 second response time [13:54:24] PROBLEM - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:24] PROBLEM - Frontend Squid HTTP on amssq55 is CRITICAL: Connection timed out [13:54:25] New patchset: Ottomata; "Creating admins::privatedata class to group users who need access to webrequest access logs." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62150 [13:54:44] PROBLEM - Backend Squid HTTP on amssq34 is CRITICAL: Connection timed out [13:54:44] RECOVERY - Frontend Squid HTTP on amssq41 is OK: HTTP OK: HTTP/1.0 200 OK - 1415 bytes in 1.388 second response time [13:54:54] RECOVERY - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 47333 bytes in 0.905 second response time [13:55:04] RECOVERY - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 60211 bytes in 8.382 second response time [13:55:14] PROBLEM - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:14] PROBLEM - Frontend Squid HTTP on amssq45 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:15] RECOVERY - Frontend Squid HTTP on amssq55 is OK: HTTP OK: HTTP/1.0 200 OK - 567 bytes in 1.403 second response time [13:55:24] RECOVERY - LVS HTTP IPv4 on wikipedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 64400 bytes in 0.350 second response time [13:55:24] PROBLEM - Frontend Squid HTTP on amssq62 is CRITICAL: Connection timed out [13:55:24] RECOVERY - LVS HTTP IPv4 on foundation-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 40489 bytes in 0.639 second response time [13:55:34] RECOVERY - Backend Squid HTTP on amssq34 is OK: HTTP OK: HTTP/1.0 200 OK - 1423 bytes in 0.175 second response time [13:55:34] PROBLEM - Backend Squid HTTP on amssq52 is CRITICAL: Connection timed out [13:56:04] RECOVERY - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 74074 bytes in 5.331 second response time [13:56:14] RECOVERY - Frontend Squid HTTP on amssq32 is OK: HTTP OK: HTTP/1.0 200 OK - 1418 bytes in 5.431 second response time [13:56:14] RECOVERY - Frontend Squid HTTP on amssq45 is OK: HTTP OK: HTTP/1.0 200 OK - 1418 bytes in 6.394 second response time [13:56:24] RECOVERY - Frontend Squid HTTP on amssq62 is OK: HTTP OK: HTTP/1.0 200 OK - 567 bytes in 0.180 second response time [13:56:24] PROBLEM - LVS HTTP IPv4 on wikimedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:26] PROBLEM - LVS HTTP IPv4 on wikisource-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:35] RECOVERY - Backend Squid HTTP on amssq52 is OK: HTTP OK: HTTP/1.0 200 OK - 663 bytes in 0.176 second response time [13:56:54] PROBLEM - LVS HTTPS IPv4 on wiktionary-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:59] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62150 [13:57:24] RECOVERY - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 53548 bytes in 8.362 second response time [13:57:24] PROBLEM - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:27] mark: everything okay? [13:58:24] PROBLEM - LVS HTTPS IPv4 on wikipedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:34] PROBLEM - LVS HTTP IPv4 on foundation-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:44] PROBLEM - SSH on amssq41 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:14] RECOVERY - LVS HTTP IPv4 on wikimedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 89113 bytes in 1.578 second response time [13:59:16] RECOVERY - LVS HTTP IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 49535 bytes in 4.362 second response time [13:59:16] PROBLEM - SSH on amssq57 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:24] RECOVERY - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 89389 bytes in 8.399 second response time [13:59:38] PROBLEM - LVS HTTP IPv4 on wikipedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:38] PROBLEM - SSH on amssq37 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:59:44] PROBLEM - Backend Squid HTTP on amssq33 is CRITICAL: Connection timed out [13:59:44] PROBLEM - SSH on amssq56 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:00:04] PROBLEM - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:00:14] RECOVERY - SSH on amssq57 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:00:14] RECOVERY - LVS HTTPS IPv4 on wikipedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 64852 bytes in 0.720 second response time [14:00:34] RECOVERY - Backend Squid HTTP on amssq33 is OK: HTTP OK: HTTP/1.0 200 OK - 1423 bytes in 0.464 second response time [14:00:34] RECOVERY - SSH on amssq37 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:00:34] RECOVERY - SSH on amssq41 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:00:37] is ok now [14:00:44] RECOVERY - SSH on amssq56 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:00:54] PROBLEM - Frontend Squid HTTP on amssq52 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:00:54] PROBLEM - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:04] RECOVERY - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 47104 bytes in 8.320 second response time [14:01:14] PROBLEM - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:24] RECOVERY - LVS HTTP IPv4 on wikipedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 64400 bytes in 0.349 second response time [14:01:34] PROBLEM - SSH on amssq50 is CRITICAL: Connection timed out [14:01:44] RECOVERY - Frontend Squid HTTP on amssq52 is OK: HTTP OK: HTTP/1.0 200 OK - 567 bytes in 1.040 second response time [14:01:44] RECOVERY - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 50010 bytes in 0.613 second response time [14:01:44] PROBLEM - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:54] PROBLEM - LVS HTTP IPv4 on wikibooks-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:01:54] PROBLEM - LVS HTTP IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:02:04] PROBLEM - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:02:44] PROBLEM - Frontend Squid HTTP on amssq59 is CRITICAL: Connection timed out [14:02:44] RECOVERY - LVS HTTP IPv4 on wikibooks-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 46861 bytes in 3.349 second response time [14:03:24] PROBLEM - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:24] PROBLEM - SSH on amssq46 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:54] RECOVERY - LVS HTTPS IPv4 on wiktionary-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 68985 bytes in 8.427 second response time [14:04:14] RECOVERY - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 73887 bytes in 8.327 second response time [14:04:44] RECOVERY - Frontend Squid HTTP on amssq59 is OK: HTTP OK: HTTP/1.0 200 OK - 567 bytes in 8.493 second response time [14:04:54] PROBLEM - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:25] PROBLEM - SSH on amssq51 is CRITICAL: Connection timed out [14:05:25] RECOVERY - LVS HTTP IPv4 on foundation-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 40487 bytes in 0.350 second response time [14:05:25] RECOVERY - SSH on amssq50 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:05:45] PROBLEM - SSH on amssq45 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:05:45] RECOVERY - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 50005 bytes in 5.163 second response time [14:05:55] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [14:05:55] PROBLEM - LVS HTTP IPv4 on wikibooks-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:05] PROBLEM - LVS HTTP IPv4 on wikimedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:15] PROBLEM - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:06:15] PROBLEM - Frontend Squid HTTP on amssq55 is CRITICAL: Connection timed out [14:06:25] RECOVERY - SSH on amssq51 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:06:35] RECOVERY - SSH on amssq46 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:06:35] RECOVERY - SSH on amssq45 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:06:35] RECOVERY - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 40857 bytes in 0.608 second response time [14:06:45] RECOVERY - LVS HTTP IPv4 on wikibooks-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 46861 bytes in 3.862 second response time [14:06:55] RECOVERY - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 60211 bytes in 8.402 second response time [14:07:45] PROBLEM - SSH on amssq44 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:07:55] RECOVERY - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 53738 bytes in 1.254 second response time [14:08:15] PROBLEM - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:08:35] RECOVERY - SSH on amssq44 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:08:35] RECOVERY - LVS HTTP IPv4 on wiktionary-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 69220 bytes in 9.976 second response time [14:08:35] PROBLEM - LVS HTTP IPv4 on foundation-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:08:45] PROBLEM - SSH on amssq35 is CRITICAL: Connection timed out [14:09:35] RECOVERY - SSH on amssq35 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:09:35] PROBLEM - Frontend Squid HTTP on amssq40 is CRITICAL: Connection timed out [14:09:45] PROBLEM - Frontend Squid HTTP on amssq60 is CRITICAL: Connection timed out [14:09:55] PROBLEM - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:45] PROBLEM - SSH on amssq56 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:45] PROBLEM - SSH on amssq31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:45] PROBLEM - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:45] RECOVERY - LVS HTTP IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 59927 bytes in 4.954 second response time [14:10:55] PROBLEM - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:55] PROBLEM - LVS HTTPS IPv4 on wiktionary-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:55] RECOVERY - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 60211 bytes in 8.294 second response time [14:10:55] RECOVERY - LVS HTTP IPv4 on wikimedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 89114 bytes in 0.720 second response time [14:11:15] PROBLEM - LVS HTTP IPv4 on wikisource-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:11:35] RECOVERY - Frontend Squid HTTP on amssq40 is OK: HTTP OK: HTTP/1.0 200 OK - 1415 bytes in 1.415 second response time [14:11:35] PROBLEM - LVS HTTP IPv4 on wiktionary-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:11:35] RECOVERY - SSH on amssq31 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:11:45] PROBLEM - SSH on ms-fe3001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:11:45] RECOVERY - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 40854 bytes in 4.935 second response time [14:11:45] RECOVERY - Frontend Squid HTTP on amssq60 is OK: HTTP OK: HTTP/1.0 200 OK - 567 bytes in 4.346 second response time [14:11:55] RECOVERY - LVS HTTP IPv4 on wikinews-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 73612 bytes in 2.523 second response time [14:11:55] PROBLEM - SSH on amssq49 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:05] RECOVERY - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 89577 bytes in 1.023 second response time [14:12:15] PROBLEM - Frontend Squid HTTP on amssq32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:15] RECOVERY - Frontend Squid HTTP on amssq55 is OK: HTTP OK: HTTP/1.0 200 OK - 565 bytes in 5.725 second response time [14:12:35] PROBLEM - Frontend Squid HTTP on amssq31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:35] RECOVERY - SSH on ms-fe3001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [14:12:45] RECOVERY - LVS HTTPS IPv4 on wiktionary-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 69692 bytes in 5.174 second response time [14:12:55] RECOVERY - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 49818 bytes in 8.325 second response time [14:12:55] RECOVERY - SSH on amssq49 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:13:05] RECOVERY - Frontend Squid HTTP on amssq32 is OK: HTTP OK: HTTP/1.0 200 OK - 1414 bytes in 2.059 second response time [14:13:15] PROBLEM - SSH on amssq36 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:15] RECOVERY - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 73887 bytes in 8.484 second response time [14:13:25] RECOVERY - Frontend Squid HTTP on amssq31 is OK: HTTP OK: HTTP/1.0 200 OK - 1414 bytes in 1.183 second response time [14:13:35] PROBLEM - SSH on amssq39 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:45] PROBLEM - SSH on amssq61 is CRITICAL: Connection timed out [14:13:55] PROBLEM - LVS HTTP IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:14:05] PROBLEM - LVS HTTP IPv4 on wikimedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:14:07] RECOVERY - SSH on amssq36 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:14:25] PROBLEM - SSH on amssq58 is CRITICAL: Connection timed out [14:14:35] RECOVERY - SSH on amssq56 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:14:35] PROBLEM - Backend Squid HTTP on amssq46 is CRITICAL: Connection timed out [14:15:05] PROBLEM - LVS HTTP IPv4 on wikinews-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:15] RECOVERY - LVS HTTP IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 49533 bytes in 8.048 second response time [14:15:15] PROBLEM - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:25] RECOVERY - SSH on amssq39 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:15:35] RECOVERY - Backend Squid HTTP on amssq46 is OK: HTTP OK: HTTP/1.0 200 OK - 1423 bytes in 3.163 second response time [14:15:45] RECOVERY - SSH on amssq61 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:15:45] PROBLEM - SSH on amssq59 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:45] RECOVERY - LVS HTTP IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 59927 bytes in 5.000 second response time [14:15:58] PROBLEM - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:16:15] PROBLEM - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:16:25] RECOVERY - SSH on amssq58 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:16:25] PROBLEM - Frontend Squid HTTP on amssq31 is CRITICAL: Connection timed out [14:16:35] PROBLEM - SSH on amssq51 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:16:35] PROBLEM - LVS HTTP IPv4 on wikipedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:16:35] RECOVERY - SSH on amssq59 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:16:45] RECOVERY - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 60402 bytes in 3.178 second response time [14:16:45] PROBLEM - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:16:45] PROBLEM - SSH on amssq38 is CRITICAL: Connection timed out [14:17:05] PROBLEM - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:17:25] RECOVERY - LVS HTTP IPv4 on wikipedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 64400 bytes in 0.350 second response time [14:17:25] RECOVERY - Frontend Squid HTTP on amssq31 is OK: HTTP OK: HTTP/1.0 200 OK - 1417 bytes in 3.636 second response time [14:17:25] PROBLEM - SSH on amssq33 is CRITICAL: Connection timed out [14:17:25] PROBLEM - Frontend Squid HTTP on amssq56 is CRITICAL: Connection timed out [14:17:35] PROBLEM - SSH on amssq32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:17:46] RECOVERY - LVS HTTP IPv4 on wikiversity-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 53260 bytes in 1.901 second response time [14:17:46] RECOVERY - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 40666 bytes in 8.266 second response time [14:17:55] RECOVERY - SSH on amssq38 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:17:55] PROBLEM - SSH on amssq43 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:17:55] PROBLEM - SSH on amssq49 is CRITICAL: Connection timed out [14:18:15] PROBLEM - LVS HTTP IPv4 on wikisource-lb.esams.wikimedia.org is CRITICAL: Connection timed out [14:18:25] RECOVERY - Frontend Squid HTTP on amssq56 is OK: HTTP OK: HTTP/1.0 200 OK - 567 bytes in 3.458 second response time [14:18:25] RECOVERY - SSH on amssq33 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:18:35] PROBLEM - Frontend Squid HTTP on amssq46 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:18:45] RECOVERY - SSH on amssq43 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:18:55] PROBLEM - LVS HTTPS IPv4 on wiktionary-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:18:55] PROBLEM - LVS HTTP IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:19:25] RECOVERY - Frontend Squid HTTP on amssq46 is OK: HTTP OK: HTTP/1.0 200 OK - 1415 bytes in 2.285 second response time [14:19:35] RECOVERY - SSH on amssq51 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:19:35] PROBLEM - SSH on amssq56 is CRITICAL: Connection timed out [14:19:55] PROBLEM - LVS HTTP IPv4 on wikibooks-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:19:55] PROBLEM - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:15] PROBLEM - SSH on amssq36 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:25] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [14:20:25] PROBLEM - Frontend Squid HTTP on amssq31 is CRITICAL: Connection timed out [14:20:35] RECOVERY - SSH on amssq32 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:20:35] RECOVERY - SSH on amssq56 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:20:45] RECOVERY - SSH on amssq49 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:20:45] PROBLEM - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:55] PROBLEM - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:55] PROBLEM - LVS HTTP IPv4 on wikiversity-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:55] PROBLEM - Frontend Squid HTTP on amssq54 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:20:55] PROBLEM - Frontend Squid HTTP on amssq58 is CRITICAL: Connection timed out [14:21:05] RECOVERY - SSH on amssq36 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:21:05] RECOVERY - LVS HTTP IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 49535 bytes in 2.043 second response time [14:21:15] PROBLEM - Backend Squid HTTP on amssq55 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:21:27] PROBLEM - SSH on amssq58 is CRITICAL: Connection timed out [14:21:35] PROBLEM - SSH on amssq33 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:21:35] PROBLEM - Frontend Squid HTTP on amssq56 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:21:35] RECOVERY - LVS HTTP IPv4 on foundation-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 40486 bytes in 8.164 second response time [14:21:45] PROBLEM - SSH on amssq31 is CRITICAL: Connection timed out [14:21:45] RECOVERY - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 40856 bytes in 8.877 second response time [14:21:45] RECOVERY - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 50008 bytes in 6.853 second response time [14:21:55] RECOVERY - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 60402 bytes in 6.334 second response time [14:21:55] RECOVERY - Frontend Squid HTTP on amssq58 is OK: HTTP OK: HTTP/1.0 200 OK - 567 bytes in 4.238 second response time [14:22:05] RECOVERY - Backend Squid HTTP on amssq55 is OK: HTTP OK: HTTP/1.0 200 OK - 663 bytes in 2.229 second response time [14:22:15] RECOVERY - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 73887 bytes in 8.410 second response time [14:22:25] RECOVERY - SSH on amssq33 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:22:25] PROBLEM - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:35] PROBLEM - SSH on amssq34 is CRITICAL: Connection timed out [14:22:35] RECOVERY - SSH on amssq31 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:22:55] RECOVERY - Frontend Squid HTTP on amssq54 is OK: HTTP OK: HTTP/1.0 200 OK - 565 bytes in 9.618 second response time [14:23:05] PROBLEM - SSH on amssq53 is CRITICAL: Connection timed out [14:23:15] PROBLEM - Frontend Squid HTTP on amssq32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:15] RECOVERY - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 47121 bytes in 1.138 second response time [14:23:25] RECOVERY - SSH on amssq58 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:23:25] RECOVERY - Frontend Squid HTTP on amssq56 is OK: HTTP OK: HTTP/1.0 200 OK - 567 bytes in 1.800 second response time [14:23:25] PROBLEM - SSH on amssq39 is CRITICAL: Connection timed out [14:23:35] PROBLEM - SSH on amssq32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:45] PROBLEM - SSH on amssq55 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:45] RECOVERY - LVS HTTP IPv4 on wikibooks-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 46862 bytes in 3.962 second response time [14:24:05] RECOVERY - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 89577 bytes in 1.074 second response time [14:24:15] PROBLEM - LVS HTTP IPv4 on wikisource-lb.esams.wikimedia.org is CRITICAL: Connection timed out [14:24:15] PROBLEM - Recursive DNS on 91.198.174.6 is CRITICAL: CRITICAL - Plugin timed out while executing system call [14:24:25] RECOVERY - SSH on amssq39 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:24:25] RECOVERY - SSH on amssq34 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:24:25] PROBLEM - SSH on amssq52 is CRITICAL: Connection timed out [14:24:35] PROBLEM - LVS HTTP IPv4 on foundation-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:24:35] RECOVERY - mysqld processes on db74 is OK: PROCS OK: 1 process with command name mysqld [14:24:55] RECOVERY - LVS HTTPS IPv4 on wiktionary-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 68985 bytes in 8.424 second response time [14:24:55] RECOVERY - SSH on amssq53 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:24:55] RECOVERY - LVS HTTP IPv4 on wikinews-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 73610 bytes in 0.746 second response time [14:25:05] RECOVERY - Recursive DNS on 91.198.174.6 is OK: DNS OK: 0.093 seconds response time. www.wikipedia.org returns 91.198.174.225 [14:25:15] PROBLEM - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:25] RECOVERY - SSH on amssq52 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:25:35] RECOVERY - SSH on amssq32 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:25:35] RECOVERY - SSH on amssq55 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:25:35] PROBLEM - SSH on amssq45 is CRITICAL: Connection timed out [14:25:55] PROBLEM - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:26:56] PROBLEM - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:26:56] RECOVERY - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 60211 bytes in 8.403 second response time [14:26:56] RECOVERY - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 53738 bytes in 4.166 second response time [14:27:05] PROBLEM - SSH on amssq47 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:27:26] RECOVERY - LVS HTTP IPv4 on wiktionary-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 69218 bytes in 5.357 second response time [14:27:35] RECOVERY - SSH on amssq45 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:27:35] RECOVERY - LVS HTTP IPv4 on foundation-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 40488 bytes in 6.207 second response time [14:27:45] RECOVERY - LVS HTTP IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 59930 bytes in 1.862 second response time [14:27:55] RECOVERY - SSH on amssq47 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:27:55] PROBLEM - LVS HTTPS IPv4 on wiktionary-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:28:05] PROBLEM - LVS HTTP IPv4 on wikinews-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:28:25] RECOVERY - Frontend Squid HTTP on amssq31 is OK: HTTP OK: HTTP/1.0 200 OK - 1413 bytes in 2.037 second response time [14:28:45] PROBLEM - SSH on amssq55 is CRITICAL: Connection timed out [14:28:55] RECOVERY - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 49818 bytes in 8.331 second response time [14:28:55] RECOVERY - LVS HTTP IPv4 on wikimedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 89114 bytes in 4.626 second response time [14:29:15] RECOVERY - Frontend Squid HTTP on amssq32 is OK: HTTP OK: HTTP/1.0 200 OK - 1415 bytes in 8.739 second response time [14:29:25] PROBLEM - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:29:35] PROBLEM - SSH on amssq32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:29:35] PROBLEM - Backend Squid HTTP on amssq60 is CRITICAL: Connection timed out [14:29:45] PROBLEM - SSH on amssq31 is CRITICAL: Connection timed out [14:29:45] PROBLEM - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:29:55] PROBLEM - LVS HTTP IPv4 on wikibooks-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:29:55] PROBLEM - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:30:15] RECOVERY - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 47335 bytes in 4.058 second response time [14:30:25] RECOVERY - SSH on amssq32 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:30:35] PROBLEM - LVS HTTP IPv4 on wiktionary-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:30:35] RECOVERY - Backend Squid HTTP on amssq60 is OK: HTTP OK: HTTP/1.0 200 OK - 663 bytes in 5.220 second response time [14:30:45] RECOVERY - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 40666 bytes in 8.265 second response time [14:30:55] PROBLEM - Frontend Squid HTTP on amssq58 is CRITICAL: Connection timed out [14:31:05] RECOVERY - LVS HTTP IPv4 on wikinews-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 73608 bytes in 8.536 second response time [14:31:05] PROBLEM - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:35] RECOVERY - SSH on amssq55 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:31:45] RECOVERY - SSH on amssq31 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:31:45] RECOVERY - LVS HTTP IPv4 on wikibooks-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 46651 bytes in 0.715 second response time [14:31:45] RECOVERY - LVS HTTP IPv4 on wikiversity-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 53262 bytes in 1.200 second response time [14:31:55] PROBLEM - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:55] RECOVERY - Frontend Squid HTTP on amssq58 is OK: HTTP OK: HTTP/1.0 200 OK - 567 bytes in 1.176 second response time [14:31:55] PROBLEM - LVS HTTP IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:05] PROBLEM - LVS HTTP IPv4 on wikimedia-lb.esams.wikimedia.org is CRITICAL: Connection timed out [14:32:07] PROBLEM - Frontend Squid HTTP on amssq35 is CRITICAL: Connection timed out [14:32:15] PROBLEM - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:35] PROBLEM - LVS HTTP IPv4 on foundation-lb.esams.wikimedia.org is CRITICAL: Connection timed out [14:32:35] PROBLEM - SSH on amssq59 is CRITICAL: Connection timed out [14:32:45] RECOVERY - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 60402 bytes in 1.082 second response time [14:32:55] RECOVERY - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 53738 bytes in 1.251 second response time [14:33:05] RECOVERY - Frontend Squid HTTP on amssq35 is OK: HTTP OK: HTTP/1.0 200 OK - 1416 bytes in 6.728 second response time [14:33:05] PROBLEM - LVS HTTPS IPv4 on wikipedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:33:15] PROBLEM - Frontend Squid HTTP on amssq45 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:33:25] PROBLEM - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:33:25] RECOVERY - LVS HTTP IPv4 on wiktionary-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 69223 bytes in 2.082 second response time [14:33:25] PROBLEM - Backend Squid HTTP on amssq61 is CRITICAL: Connection timed out [14:33:45] PROBLEM - Frontend Squid HTTP on amssq33 is CRITICAL: Connection timed out [14:33:45] RECOVERY - LVS HTTP IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 59930 bytes in 3.867 second response time [14:33:45] PROBLEM - SSH on amssq38 is CRITICAL: Connection timed out [14:34:05] PROBLEM - LVS HTTP IPv4 on wikinews-lb.esams.wikimedia.org is CRITICAL: Connection timed out [14:34:25] RECOVERY - Backend Squid HTTP on amssq61 is OK: HTTP OK: HTTP/1.0 200 OK - 663 bytes in 0.175 second response time [14:34:35] PROBLEM - SSH on amssq34 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:34:35] RECOVERY - SSH on amssq59 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:34:35] RECOVERY - Frontend Squid HTTP on amssq33 is OK: HTTP OK: HTTP/1.0 200 OK - 1413 bytes in 0.179 second response time [14:35:04] PROBLEM - Frontend Squid HTTP on amssq54 is CRITICAL: Connection timed out [14:35:04] RECOVERY - LVS HTTP IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 49535 bytes in 0.378 second response time [14:35:14] RECOVERY - Frontend Squid HTTP on amssq45 is OK: HTTP OK: HTTP/1.0 200 OK - 1417 bytes in 4.164 second response time [14:35:14] RECOVERY - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 50010 bytes in 0.622 second response time [14:35:23] New patchset: Brion VIBBER; "Update FirefoxOS app for manifest validation issue" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62153 [14:35:34] RECOVERY - LVS HTTP IPv4 on foundation-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 40487 bytes in 0.979 second response time [14:35:34] RECOVERY - Frontend Squid HTTP on amssq54 is OK: HTTP OK: HTTP/1.0 200 OK - 565 bytes in 6.860 second response time [14:35:54] PROBLEM - SSH on amssq43 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:36:08] are you doing maint, or is it generic fnuckup? [14:36:13] both [14:36:14] RECOVERY - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 47336 bytes in 3.835 second response time [14:36:24] PROBLEM - LVS HTTP IPv4 on wikibooks-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:36:24] PROBLEM - LVS HTTP IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:36:24] RECOVERY - LVS HTTPS IPv4 on wikipedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 64852 bytes in 0.717 second response time [14:36:44] RECOVERY - SSH on amssq43 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:36:50] brion left the waffles on the server again? [14:37:34] PROBLEM - SSH on amssq37 is CRITICAL: Connection timed out [14:38:14] RECOVERY - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 73856 bytes in 8.445 second response time [14:38:24] RECOVERY - SSH on amssq38 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:38:24] PROBLEM - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:38:24] RECOVERY - SSH on amssq34 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:38:34] PROBLEM - LVS HTTP IPv4 on wikipedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:38:44] PROBLEM - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:39:14] PROBLEM - LVS HTTP IPv4 on wikisource-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:39:14] PROBLEM - Host amssq57 is DOWN: PING CRITICAL - Packet loss = 100% [14:39:24] RECOVERY - LVS HTTPS IPv4 on wiktionary-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 68985 bytes in 8.315 second response time [14:39:34] RECOVERY - SSH on amssq37 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:39:34] PROBLEM - Host amssq58 is DOWN: PING CRITICAL - Packet loss = 100% [14:39:54] PROBLEM - Host amssq56 is DOWN: PING CRITICAL - Packet loss = 100% [14:39:54] PROBLEM - Host professor is DOWN: PING CRITICAL - Packet loss = 100% [14:40:04] PROBLEM - Host amssq49 is DOWN: PING CRITICAL - Packet loss = 100% [14:40:04] PROBLEM - Host knsq19 is DOWN: PING CRITICAL - Packet loss = 100% [14:40:04] PROBLEM - Host amssq61 is DOWN: PING CRITICAL - Packet loss = 100% [14:40:14] PROBLEM - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:14] PROBLEM - Host cp3003 is DOWN: PING CRITICAL - Packet loss = 100% [14:40:14] PROBLEM - Host amssq50 is DOWN: PING CRITICAL - Packet loss = 100% [14:40:14] PROBLEM - Host maerlant is DOWN: PING CRITICAL - Packet loss = 100% [14:40:23] if nobody else is awake should i poke RobH ? [14:40:24] RECOVERY - LVS HTTP IPv4 on wikibooks-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 46858 bytes in 7.823 second response time [14:40:24] PROBLEM - Host knsq18 is DOWN: PING CRITICAL - Packet loss = 100% [14:40:32] what's with morebots / Server Admin Log or is it related to esams-issue? [14:40:34] PROBLEM - LVS HTTPS IPv4 on wikipedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:40:34] PROBLEM - Host amssq53 is DOWN: PING CRITICAL - Packet loss = 100% [14:40:37] mark is "awake" [14:41:00] se4598: it died in a netsplit [14:41:02] brion: It's "fine" [14:41:02] se4598: looks like it netsplitted [14:41:04] PROBLEM - LVS HTTP IPv4 on wikiversity-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:41:04] PROBLEM - Host ms-fe3001 is DOWN: PING CRITICAL - Packet loss = 100% [14:41:05] heh [14:41:14] PROBLEM - Host ms6 is DOWN: PING CRITICAL - Packet loss = 100% [14:41:24] RECOVERY - LVS HTTPS IPv4 on wikipedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 64879 bytes in 0.708 second response time [14:41:25] PROBLEM - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:41:34] PROBLEM - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:41:44] PROBLEM - SSH on amssq45 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:41:50] New patchset: Ottomata; "Adding mwalker on stat1 and stat1002 (admins::privatedata)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62154 [14:41:54] PROBLEM - Host amssq60 is DOWN: PING CRITICAL - Packet loss = 100% [14:42:10] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62154 [14:42:14] PROBLEM - Host amssq62 is DOWN: PING CRITICAL - Packet loss = 100% [14:42:24] RECOVERY - LVS HTTP IPv4 on wikipedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 64396 bytes in 0.350 second response time [14:42:34] PROBLEM - SSH on amssq39 is CRITICAL: Connection timed out [14:42:34] RECOVERY - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 60211 bytes in 8.384 second response time [14:42:35] PROBLEM - LVS HTTP IPv4 on foundation-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:42:44] PROBLEM - Host amssq52 is DOWN: PING CRITICAL - Packet loss = 100% [14:43:14] RECOVERY - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 89574 bytes in 5.262 second response time [14:43:14] PROBLEM - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:43:24] RECOVERY - LVS HTTP IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 59929 bytes in 9.790 second response time [14:43:24] PROBLEM - LVS HTTP IPv4 on wikibooks-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:43:24] RECOVERY - SSH on amssq39 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:43:34] PROBLEM - LVS HTTP IPv4 on wiktionary-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:43:34] RECOVERY - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 40857 bytes in 5.090 second response time [14:43:44] RECOVERY - SSH on amssq45 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:43:54] PROBLEM - Host amssq48 is DOWN: PING CRITICAL - Packet loss = 100% [14:44:04] PROBLEM - Host amssq54 is DOWN: PING CRITICAL - Packet loss = 100% [14:44:04] RECOVERY - LVS HTTP IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 49535 bytes in 4.584 second response time [14:44:14] RECOVERY - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 73854 bytes in 8.426 second response time [14:44:14] RECOVERY - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 47333 bytes in 0.623 second response time [14:44:14] RECOVERY - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 50005 bytes in 4.910 second response time [14:44:24] PROBLEM - Host amssq55 is DOWN: PING CRITICAL - Packet loss = 100% [14:44:54] RECOVERY - LVS HTTP IPv4 on wikiversity-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 53260 bytes in 3.499 second response time [14:45:24] PROBLEM - Host amssq47 is DOWN: PING CRITICAL - Packet loss = 100% [14:45:24] RECOVERY - LVS HTTP IPv4 on foundation-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 40487 bytes in 0.975 second response time [14:45:34] PROBLEM - LVS HTTP IPv4 on wikipedia-lb.esams.wikimedia.org is CRITICAL: Connection timed out [14:45:34] PROBLEM - SSH on amssq33 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:45:34] PROBLEM - LVS HTTPS IPv4 on wikipedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:45:34] PROBLEM - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:45:44] RECOVERY - LVS HTTP IPv4 on wikinews-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 73606 bytes in 5.667 second response time [14:46:14] PROBLEM - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:24] RECOVERY - SSH on amssq33 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:46:24] PROBLEM - LVS HTTP IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:24] PROBLEM - Host knsq17 is DOWN: PING CRITICAL - Packet loss = 100% [14:47:14] RECOVERY - LVS HTTP IPv4 on wikibooks-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 46858 bytes in 0.431 second response time [14:47:14] PROBLEM - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:14] PROBLEM - LVS HTTP IPv4 on wikisource-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:15] anyone working on the issue? [14:47:24] PROBLEM - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:34] RECOVERY - LVS HTTP IPv4 on wiktionary-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 69221 bytes in 8.383 second response time [14:47:40] Yes [14:47:44] PROBLEM - SSH on amssq44 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:47:44] PROBLEM - SSH on amssq31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:04] RECOVERY - LVS HTTP IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 49535 bytes in 1.847 second response time [14:48:14] RECOVERY - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 73855 bytes in 8.415 second response time [14:48:14] PROBLEM - Host amssq51 is DOWN: PING CRITICAL - Packet loss = 100% [14:48:14] PROBLEM - Host knsq20 is DOWN: PING CRITICAL - Packet loss = 100% [14:48:24] RECOVERY - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 49818 bytes in 8.332 second response time [14:48:24] RECOVERY - LVS HTTP IPv4 on wikipedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 64396 bytes in 0.356 second response time [14:48:34] RECOVERY - SSH on amssq31 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:48:44] PROBLEM - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:44] RECOVERY - SSH on amssq44 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [14:48:54] PROBLEM - LVS HTTP IPv4 on wikinews-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:49:24] RECOVERY - LVS HTTPS IPv4 on wikipedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 64879 bytes in 0.706 second response time [14:49:24] PROBLEM - Host nescio is DOWN: PING CRITICAL - Packet loss = 100% [14:49:24] PROBLEM - Host ns2.wikimedia.org is DOWN: PING CRITICAL - Packet loss = 100% [14:49:44] PROBLEM - Host amssq59 is DOWN: PING CRITICAL - Packet loss = 100% [14:49:54] PROBLEM - Host knsq22 is DOWN: PING CRITICAL - Packet loss = 100% [14:49:54] PROBLEM - Host 91.198.174.6 is DOWN: PING CRITICAL - Packet loss = 100% [14:50:04] PROBLEM - LVS HTTP IPv4 on wikiversity-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:50:04] RECOVERY - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 89580 bytes in 1.012 second response time [14:50:14] RECOVERY - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 53548 bytes in 8.360 second response time [14:50:24] PROBLEM - LVS HTTPS IPv4 on wiktionary-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:50:34] RECOVERY - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 60211 bytes in 8.282 second response time [14:50:34] RECOVERY - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 40857 bytes in 1.356 second response time [14:50:35] RECOVERY - Host knsq19 is UP: PING OK - Packet loss = 0%, RTA = 89.09 ms [14:50:35] RECOVERY - LVS HTTP IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64855 bytes in 0.391 second response time [14:50:35] RECOVERY - Host knsq20 is UP: PING OK - Packet loss = 0%, RTA = 86.66 ms [14:50:35] RECOVERY - Host knsq22 is UP: PING OK - Packet loss = 0%, RTA = 90.43 ms [14:50:35] RECOVERY - LVS HTTP IPv4 on wikimedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 89114 bytes in 0.534 second response time [14:50:37] RECOVERY - LVS HTTPS IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64855 bytes in 0.707 second response time [14:50:37] RECOVERY - LVS HTTPS IPv6 on foundation-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64855 bytes in 0.705 second response time [14:50:44] RECOVERY - LVS HTTPS IPv6 on bits-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 3917 bytes in 0.440 second response time [14:50:46] RECOVERY - Host knsq18 is UP: PING OK - Packet loss = 0%, RTA = 86.78 ms [14:50:46] RECOVERY - LVS HTTP IPv4 on wikinews-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 73608 bytes in 0.437 second response time [14:50:46] RECOVERY - LVS HTTPS IPv6 on wikipedia-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64855 bytes in 0.701 second response time [14:50:47] RECOVERY - LVS HTTPS IPv6 on wikimedia-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 89577 bytes in 0.798 second response time [14:50:47] RECOVERY - Host knsq17 is UP: PING OK - Packet loss = 0%, RTA = 88.47 ms [14:50:54] RECOVERY - LVS HTTP IPv6 on upload-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 562 bytes in 0.176 second response time [14:50:55] RECOVERY - LVS HTTP IPv4 on wikiversity-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 53260 bytes in 0.350 second response time [14:51:04] RECOVERY - LVS HTTPS IPv6 on upload-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 560 bytes in 0.449 second response time [14:51:06] RECOVERY - LVS HTTPS IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64883 bytes in 0.706 second response time [14:51:06] RECOVERY - LVS HTTPS IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64855 bytes in 0.708 second response time [14:51:06] RECOVERY - LVS HTTPS IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64855 bytes in 0.697 second response time [14:51:14] RECOVERY - LVS HTTP IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.0 200 OK - 59932 bytes in 0.350 second response time [14:51:14] RECOVERY - LVS HTTP IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64855 bytes in 0.349 second response time [14:51:14] RECOVERY - LVS HTTPS IPv4 on wiktionary-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 69697 bytes in 0.886 second response time [14:51:14] RECOVERY - LVS HTTP IPv6 on wikinews-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64882 bytes in 0.346 second response time [14:51:14] RECOVERY - LVS HTTP IPv6 on wikimedia-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 89577 bytes in 0.437 second response time [14:51:14] RECOVERY - LVS HTTPS IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64879 bytes in 0.705 second response time [14:51:24] RECOVERY - LVS HTTP IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64855 bytes in 0.346 second response time [14:51:24] RECOVERY - LVS HTTP IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64855 bytes in 0.350 second response time [14:51:24] RECOVERY - LVS HTTP IPv6 on foundation-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64855 bytes in 0.350 second response time [14:51:24] RECOVERY - LVS HTTPS IPv6 on wikinews-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64855 bytes in 0.700 second response time [14:51:24] RECOVERY - LVS HTTPS IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64882 bytes in 0.705 second response time [14:51:24] RECOVERY - LVS HTTP IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64855 bytes in 0.349 second response time [14:51:34] RECOVERY - LVS HTTP IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64879 bytes in 0.349 second response time [14:51:35] RECOVERY - LVS HTTP IPv6 on wikipedia-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 64855 bytes in 0.387 second response time [14:52:09] stuff is all back on csw1-esams [14:52:19] !log Migrated VRRP for vlan 100 back to csw1-esams [14:52:29] mark: morebots is dead [14:56:06] ottomata: morebots is afk [14:57:29] k one sec [15:05:01] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [15:07:25] wb [16:04:59] Thehelpfulone: was going to ask if you got an interwiki map sync for wm2014 but i see it's already working :) needs another and see also RT 5071 [16:06:40] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [16:07:17] ah okay jeremyb_, reading that ticket now [16:07:50] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 16:07:44 UTC 2013 [16:08:40] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [16:08:46] icinga-wm: quiet! [16:08:49] jeremyb_, hmm, that one needs a public bugzilla ticket I think too (to make it easier for non-ops people to find it) [16:08:50] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 16:08:46 UTC 2013 [16:09:00] if we're creating a new chapter wiki [16:09:15] Thehelpfulone: maybe... it originated with non-bugzilla peoples though :) (tle) [16:09:40] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:44] *nod* but I'd still like to put it in bugzilla for the mortals ;) [16:09:50] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 16:09:47 UTC 2013 [16:10:22] Thehelpfulone: also, idk if the alternate domains would need approval (i.e. if the existing approval is for $country.wikimedia.org only) [16:10:40] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [16:10:40] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 16:10:39 UTC 2013 [16:10:50] damnit icinga-wm. go away! [16:11:03] Thehelpfulone: anyway, read my comment on the RT? [16:11:40] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:10] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 16:12:07 UTC 2013 [16:12:40] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [16:12:50] RECOVERY - Puppet freshness on db10 is OK: puppet ran at Fri May 3 16:12:42 UTC 2013 [16:12:53] yeah jeremyb_ - so would we want to create a new wiki on ph.wikimedia.org (and transfer the other domains to WMF or get some redirects set up), then import content? [16:13:40] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:44] Thehelpfulone: well what would be canonical? i personally would prefer keeping the current canonical domain [16:15:00] I agree [16:23:14] New patchset: Matthias Mullie; "No article page link for dewiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62161 [16:26:55] New patchset: Krinkle; "contint: Move apache logs to readable place for localhost testing" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61997 [16:58:27] New review: Matthias Mullie; "Not to be merged before auto-archive cron is enabled" [operations/mediawiki-config] (master) C: -2; - https://gerrit.wikimedia.org/r/60803 [17:01:36] New review: MF-Warburg; "seems alright with the code & the RFC" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/62012 [17:07:08] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [17:07:17] New review: Hahc21; "Looks good." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/62012 [17:13:18] mutante: see me and Thehelpfulone above [17:13:57] mutante: i wasn't thinking they'd import a DB. but I was thinking [[special:import]] is not permitted for security reasons of some sort [17:14:20] (not by XML upload and not by transwiki from non-WMF wikis) [17:14:43] LeslieCarr, mutante, Ryan_Lane, or whoever is around, Roan just left his laptop on BART, there's an RT ticket (number 5079) for revoking his keys [17:15:01] we should use ph.wikimedia.org, because the *.chapters.wikimedia.org proposal died afaik and just makes it all more complicated [17:15:09] jeremyb_: [17:15:25] marktraceur: we all get emails for every rt ticket [17:15:32] jeremyb_: i'm not sure which import method is best, but as long as they can do it and we dont import db [17:15:41] Ryan_Lane: He requested some urgency [17:15:42] wasnt aware of security issue [17:15:44] grrr, what's wrong with iceweasel... [17:15:51] * marktraceur will note the emails for future [17:16:23] mutante: well i think that leaves us with no options remaining [17:16:28] for import [17:16:36] what is the problem with special:import [17:17:01] i have no idea [17:17:32] yea, dunno, can't we just enable it for an initial admin and then they disabled it again after being done.. [17:17:45] New patchset: Ryan Lane; "Per Roan's request, kill his key" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62169 [17:17:47] Ryan_Lane, do you think it's worth revoking Bugzilla access too (I think he's got security access) ? [17:17:52] mutante, yeah that sounds reasonable [17:17:53] jeremyb_: What are you trying to import? (my backscroll doesn't have it..) [17:17:58] New patchset: BBlack; "vhtcpd - basically complete" [operations/software/varnish/vhtcpd] (master) - https://gerrit.wikimedia.org/r/60390 [17:18:07] csteipp: RT 5071 [17:18:11] csteipp, nothing yet, see https://rt.wikimedia.org/Ticket/Display.html?id=5071 [17:19:11] Thehelpfulone: nah [17:19:38] New patchset: Ryan Lane; "Per Roan's request, kill his key" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62169 [17:19:43] csteipp: the content of wikimedia.org.ph [17:19:56] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62169 [17:20:33] i need to go guys, ttyl [17:20:55] Ah. Yeah, for WV we did a db import, which will probably work a little better than special:import [17:21:28] but we would like to avoid that [17:21:37] we would like them to handle the import after we just give them a wiki [17:22:57] away [17:23:43] Ah. So yeah, you can temporarily give the import right out. The code's a little ugly, but I think it's disable just to minimize the attack surface. No current exploits in it. [17:41:13] RECOVERY - Host db1020 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [17:54:21] mark, would you please look at change 62105? this is sort of urgent, and awjr is cool with getting it deployed because we've isolated it to only impact Wikipedia Zero requests. also sent an e-mail with you and greg-g on it. greg-g said he's cool with pushing this out today if you are. [17:57:55] dr0ptp4kt: i think mark's been kinda busy today... [17:58:35] jeremyb_: who else should dr0ptp4kt ping re varnish stuff? [17:58:45] maybe asher? [17:58:54] greg-g: idk, i'm reading the patch first [17:58:58] :) [18:00:00] well, small diffstat at least :) [18:00:42] greg-g: i guess first i'd try bblack. (skipping para void because it's already 9pm on a friday night) [18:01:02] awjr: dr0ptp4kt: ^ [18:01:23] :) [18:04:15] varnish stuff is probably mark or asher, yeah [18:04:48] or bblack or faidon :) [18:05:10] paravoid: could you merge https://gerrit.wikimedia.org/r/#/c/54116/ ? [18:05:43] drdee: it still needs a solution for the recommended package stuff [18:05:47] i can work on that today [18:05:55] but faidon indicated that this was a blocker [18:09:23] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [18:15:39] dr0ptp4kt: +1 [18:15:53] PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours [18:21:23] New patchset: Odder; "(bug 45696) Add a msg to $wgForceUIMsgAsContentMsg for Wikidata" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62175 [18:28:37] Reedy, when you login/log out of wikimania 2014 wiki, all the logos for the sister projects show where they're supposed to link to as a tooltip, but don't actually link to the other wikis, any idea what could be causing that? [18:31:14] Thehelpfulone: errr, where would you like them to link to? [18:31:54] * jeremyb_ just compared with meta and it also doesn't have them [18:31:59] oh, hmm [18:32:12] well I'd have thought each wiki would be good? [18:32:28] so for wikipedia it'd be wikipedia.org or en.wikipedia if you can do something fancy with the language [18:32:52] IMHo it's better if they don't link anywhere, there's no purpose in clicking [18:33:12] oh another thing jeremyb_ I just checked on enwiki, wikimania doesn't show up there - did the logo show up on meta? [18:33:30] idk [18:33:50] i think i agree with nemo [18:34:29] i see no wikimania.org on meta [18:49:24] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [18:53:14] wikimania is not in autologin [19:05:31] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [19:28:05] New patchset: Pyoungmeister; "adding db74 to db-pmtpa.php" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62184 [19:53:55] !log maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/includes/MobileFrontend.hooks.php [19:54:03] Logged the message, Master [20:03:48] !log maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/MobileFrontend.hooks.php [20:03:56] Logged the message, Master [20:09:29] binasher: is professor down again? [20:10:25] AaronSchulz: was down yesterday [20:10:27] so it seems [20:10:30] probably still down today [20:10:53] hrmmm, https://rt.wikimedia.org/Ticket/Display.html?id=5068 [20:11:12] it was up for a while yesterday or something (maybe the day before) [20:11:34] it's basicallly like wikipedia's own Cultural Revolution [20:11:45] something wrong with anon icinga? [20:11:47] > Error: Could not read host and service status information! [20:12:29] AaronSchulz: not really much we can do right now [20:13:10] i'm going to package up a newer version of graphite and migrate services to eqiad [20:13:11] binasher: see, it's that spirit of hopelessness that will get you through the dark times [20:13:48] notpeter: {{{true|that}}} [20:14:28] binasher: geez, use Lua [20:17:23] binasher: Did you ever get out that json out? [20:17:49] Coren: the csv i gave you yesterday wasn't ok? [20:18:06] Wait, I must have missed it then. path? [20:18:28] bastion-restricted1:/tmp/columns [20:20:19] binasher: Got it, thanks. Sorry I must have missed your note. [20:20:42] no prob, let me know if you need anything else [20:34:15] PROBLEM - Host db1020 is DOWN: PING CRITICAL - Packet loss = 100% [20:35:15] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [20:55:31] binasher: db1020 was hung up on a DIMM Module error from a previous reboot...something different than what I've seen in the past. I've cycled a couple of times and the server has come back without issue. Running some memtest now but nothing appears to be wrong. [20:56:02] cmjohnson1: odd, thanks! [20:56:09] cmjohnson1: do you know if anyone happens to be in pmtpa today? [20:56:23] i haven't heard from steve today ...so probably not [20:56:51] dang, ok [21:06:57] PROBLEM - Puppet freshness on db10 is CRITICAL: No successful Puppet run in the last 10 hours [21:09:29] !Log db10 is decommissioned...powering down for the final time [21:09:39] Logged the message, Master [21:10:07] PROBLEM - RAID on db10 is CRITICAL: Timeout while attempting connection [21:10:13] oooooh. and db9? [21:10:56] i haven't seen a decom ticket yet jeremyb_ [21:11:00] for db9 ^ [21:11:13] i just ask because they were a pair [21:11:39] yeah..i know I would assume so but ... [21:11:47] PROBLEM - Host db10 is DOWN: PING CRITICAL - Packet loss = 100% [21:13:29] mutante: ^^^ [21:22:05] RobH: professor died again and i can't power it up with "start /SYS" - can you try? [21:24:37] binasher: i am trying to powercycle for you [21:25:38] thanks [21:25:54] powering up reset /SYS worked [21:27:43] man, i tried stop /SYS and start /SYS [21:27:59] New patchset: Ottomata; "Adding Adam Wight on stat1 and stat1002. Changing his ssh key." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62196 [21:28:06] yeah ..suns are stupid [21:28:17] binasher: good to go! [21:28:58] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62196 [21:29:24] cmjohnson1: might be down again? doesn't respond to a ping [21:30:30] odd [21:31:39] it came back all the way to prompt so HDD is ok [21:35:01] 42 [21:35:01] 42 [21:36:19] [Bgrr [21:44:46] notpeter: ping [21:45:39] New patchset: Odder; "(bug 47820) Change $wgSitename and NS_PROJECT_TALK for udmwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62197 [22:05:14] preilly: sup [22:09:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:10:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [22:11:21] about to run a short maintanence script on mediawikiwiki [22:11:59] Change merged: Pyoungmeister; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62184 [22:19:02] !log py synchronized wmf-config/db-pmtpa.php 'adding new pmtpa slave' [22:19:09] Logged the message, Master [22:21:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:22:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.136 second response time [22:52:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:53:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [23:30:46] about to run a maintenance script on en.wiki [23:33:30] Gerrit down? [23:34:02] http://cl.ly/image/220q0o0B3j0n [23:34:12] server errors repeatedly [23:34:24] 'Guice provision errors: Cannot open ReviewDb [23:41:08] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [23:41:08] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [23:41:08] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [23:41:08] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [23:58:31] It feels slow.