[00:19:42] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:26] Thehelpfulone: http://wikitech.wikimedia.org/view/Password_reset [00:24:33] thanks Tim [00:27:29] oh god that looks nasty [00:29:19] I guess that In your browser, go to Special:ResetPassword on the user's main wiki. [00:29:19] requires them to be part of the global sysadmin group? [00:30:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.669 seconds [00:32:02] Thehelpfulone: errr, isn't that something anyone can do? [00:32:20] jeremyb, well I just tried it, I can't enter someone else's username apparently [00:32:50] anyone can do that [00:33:10] normally anons do it, it wouldn't really make sense to restrict it, would it? [00:33:27] oh I see, it's Special:PasswordReset [00:33:45] Special:ResetPassword redirects to Special:ChangePassword [00:33:57] fixed the doc [00:34:10] hmm, and I thought I had copy/pasted that [00:34:26] guess not [00:34:37] * jeremyb just figured that out at the same time [00:34:45] why aren't those synonymous?!?!?? [00:35:29] well, usually you just follow the link [00:36:31] hmm, If you are certain of your e-mail, but not your username, only enter your e-mail. -> so if you have two accounts with the same email (my account and my bot account), will it send two emails? (I can't test it now because I just did one) [00:40:19] !bug 34386 | Thehelpfulone [00:40:19] Thehelpfulone: https://bugzilla.wikimedia.org/34386 [00:40:44] in particular i guess comment 17 [00:41:09] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [00:41:46] thanks, heh Umherirrender was persistent [00:42:21] next week, next week, next week! [00:42:32] Susan: ^ :) [00:42:58] Heh. [00:43:04] Being persistent is often helpful. [00:55:06] PROBLEM - Puppet freshness on solr2 is CRITICAL: Puppet has not run in the last 10 hours [00:57:03] PROBLEM - Puppet freshness on vanadium is CRITICAL: Puppet has not run in the last 10 hours [01:05:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:07:06] PROBLEM - Puppet freshness on solr1003 is CRITICAL: Puppet has not run in the last 10 hours [01:07:06] PROBLEM - Puppet freshness on solr3 is CRITICAL: Puppet has not run in the last 10 hours [01:08:09] PROBLEM - Puppet freshness on solr1001 is CRITICAL: Puppet has not run in the last 10 hours [01:13:42] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 294 seconds [01:15:30] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 11 seconds [01:18:30] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.028 seconds [01:33:03] PROBLEM - Puppet freshness on tin is CRITICAL: Puppet has not run in the last 10 hours [01:35:18] PROBLEM - Host lvs1001 is DOWN: PING CRITICAL - Packet loss = 100% [01:36:12] RECOVERY - Host lvs1001 is UP: PING WARNING - Packet loss = 28%, RTA = 41.58 ms [01:37:15] PROBLEM - Host wiktionary-lb.eqiad.wikimedia.org is DOWN: PING CRITICAL - Packet loss = 100% [01:37:24] PROBLEM - LVS HTTP IPv4 on wikiversity-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:37:24] PROBLEM - LVS HTTP IPv6 on wikiquote-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:37:24] PROBLEM - LVS HTTP IPv4 on wikimedia-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:37:42] PROBLEM - LVS HTTPS IPv6 on wiktionary-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:37:51] PROBLEM - LVS HTTPS IPv4 on wikimedia-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:37:52] PROBLEM - LVS HTTP IPv6 on wikimedia-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:37:52] PROBLEM - LVS HTTP IPv4 on wikibooks-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:37:52] PROBLEM - LVS HTTPS IPv6 on wikisource-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:37:52] PROBLEM - LVS HTTP IPv4 on mediawiki-lb.eqiad.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:37:52] PROBLEM - LVS HTTPS IPv6 on wikimedia-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:38:00] PROBLEM - LVS HTTPS IPv6 on mediawiki-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:38:00] PROBLEM - LVS HTTPS IPv6 on wikiversity-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:38:09] PROBLEM - LVS HTTP IPv6 on wikisource-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:38:09] PROBLEM - LVS HTTPS IPv6 on wikiquote-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:38:26] ok, who unplugged the servers [01:38:27] PROBLEM - LVS HTTP IPv6 on foundation-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:38:27] PROBLEM - LVS HTTP IPv6 on wikibooks-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:38:36] PROBLEM - LVS HTTPS IPv6 on foundation-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:39:03] RECOVERY - LVS HTTP IPv4 on wikiversity-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 66649 bytes in 0.145 seconds [01:39:03] RECOVERY - LVS HTTP IPv6 on wikiquote-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 66270 bytes in 0.136 seconds [01:39:04] RECOVERY - LVS HTTP IPv4 on wikimedia-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 93182 bytes in 0.165 seconds [01:39:22] RECOVERY - LVS HTTPS IPv6 on wiktionary-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 66270 bytes in 0.196 seconds [01:39:30] RECOVERY - LVS HTTP IPv6 on wikimedia-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 92797 bytes in 0.136 seconds [01:39:30] RECOVERY - LVS HTTPS IPv4 on wikimedia-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.1 200 OK - 92797 bytes in 0.192 seconds [01:39:31] RECOVERY - LVS HTTP IPv4 on wikibooks-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 66649 bytes in 0.136 seconds [01:39:32] RECOVERY - LVS HTTP IPv4 on mediawiki-lb.eqiad.wikimedia.org is OK: HTTP OK HTTP/1.0 200 OK - 66651 bytes in 0.134 seconds [01:39:32] RECOVERY - LVS HTTPS IPv6 on wikisource-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 66270 bytes in 0.196 seconds [01:39:32] RECOVERY - LVS HTTPS IPv6 on wikimedia-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 92797 bytes in 0.222 seconds [01:39:39] RECOVERY - LVS HTTPS IPv6 on mediawiki-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 66270 bytes in 0.193 seconds [01:39:39] RECOVERY - LVS HTTPS IPv6 on wikiversity-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 66270 bytes in 0.193 seconds [01:39:48] RECOVERY - LVS HTTP IPv6 on wikisource-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 66272 bytes in 0.137 seconds [01:39:48] RECOVERY - LVS HTTPS IPv6 on wikiquote-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 66272 bytes in 0.193 seconds [01:40:06] RECOVERY - LVS HTTP IPv6 on foundation-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 66270 bytes in 0.138 seconds [01:40:06] RECOVERY - LVS HTTP IPv6 on wikibooks-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 66270 bytes in 0.138 seconds [01:40:06] RECOVERY - LVS HTTPS IPv6 on foundation-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK HTTP/1.1 200 OK - 66270 bytes in 0.193 seconds [01:40:15] RECOVERY - Host wiktionary-lb.eqiad.wikimedia.org is UP: PING OK - Packet loss = 0%, RTA = 27.03 ms [01:41:58] lesliecarr - recovered? [01:42:04] i think so [01:42:39] someone needs to feed the hamsters [01:45:14] PROBLEM - Host wikimedia-lb.pmtpa.wikimedia.org is DOWN: PING CRITICAL - Packet loss = 100% [01:45:58] doh, i fed the hamster poison instead of hamster food [01:46:04] i knew i shouldn't have put them next to each other [01:46:05] ;) [01:46:25] RECOVERY - Host wikimedia-lb.pmtpa.wikimedia.org is UP: PING OK - Packet loss = 0%, RTA = 0.32 ms [01:52:06] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:54:30] RECOVERY - Memcached on mc1011 is OK: TCP OK - 0.027 second response time on port 11211 [01:54:30] RECOVERY - Memcached on mc1003 is OK: TCP OK - 0.028 second response time on port 11211 [01:54:39] RECOVERY - Memcached on mc1005 is OK: TCP OK - 0.028 second response time on port 11211 [01:54:48] RECOVERY - Memcached on mc1004 is OK: TCP OK - 0.030 second response time on port 11211 [01:54:48] RECOVERY - Memcached on mc1007 is OK: TCP OK - 0.028 second response time on port 11211 [01:54:57] RECOVERY - Memcached on mc1009 is OK: TCP OK - 0.027 second response time on port 11211 [01:55:06] RECOVERY - Memcached on mc1015 is OK: TCP OK - 0.027 second response time on port 11211 [01:55:06] PROBLEM - Puppet freshness on brewster is CRITICAL: Puppet has not run in the last 10 hours [01:55:15] RECOVERY - Memcached on mc1013 is OK: TCP OK - 0.027 second response time on port 11211 [01:55:15] RECOVERY - Memcached on mc1016 is OK: TCP OK - 0.030 second response time on port 11211 [01:55:33] RECOVERY - Memcached on mc1014 is OK: TCP OK - 0.027 second response time on port 11211 [01:55:33] RECOVERY - Memcached on mc1010 is OK: TCP OK - 0.027 second response time on port 11211 [01:55:51] RECOVERY - Memcached on mc1006 is OK: TCP OK - 0.027 second response time on port 11211 [01:55:51] RECOVERY - Memcached on mc1008 is OK: TCP OK - 0.027 second response time on port 11211 [01:58:35] wee, that's enough work for now. unless the pager tells me otherwise. [02:01:06] PROBLEM - Puppet freshness on sockpuppet is CRITICAL: Puppet has not run in the last 10 hours [02:06:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.027 seconds [02:19:55] did you !log the poison? [02:26:12] !log LocalisationUpdate completed (1.21wmf6) at Mon Dec 31 02:26:11 UTC 2012 [02:26:22] Logged the message, Master [02:50:26] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [02:53:35] RECOVERY - Puppet freshness on neon is OK: puppet ran at Mon Dec 31 02:53:03 UTC 2012 [02:54:20] RECOVERY - Puppet freshness on sockpuppet is OK: puppet ran at Mon Dec 31 02:54:07 UTC 2012 [02:59:26] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours [03:06:47] RECOVERY - Puppet freshness on tin is OK: puppet ran at Mon Dec 31 03:06:37 UTC 2012 [03:36:20] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [03:46:05] PROBLEM - Memcached on virt0 is CRITICAL: Connection refused [03:46:25] Ryan_Lane: virt0 memcache still bouncing ^ [04:02:53] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:06:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.582 seconds [04:07:05] RECOVERY - Memcached on virt0 is OK: TCP OK - 0.002 second response time on port 11000 [04:22:23] PROBLEM - Puppet freshness on stat1 is CRITICAL: Puppet has not run in the last 10 hours [04:40:05] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:54:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.522 seconds [05:02:44] PROBLEM - Memcached on mc1012 is CRITICAL: Connection refused [05:08:26] PROBLEM - Puppet freshness on silver is CRITICAL: Puppet has not run in the last 10 hours [05:08:26] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [05:29:44] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:40:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.140 seconds [06:15:47] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:26:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.858 seconds [06:57:14] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [07:01:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:05:20] PROBLEM - Puppet freshness on analytics1007 is CRITICAL: Puppet has not run in the last 10 hours [07:05:20] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: Puppet has not run in the last 10 hours [07:05:20] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [07:05:20] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [07:13:53] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.042 seconds [07:47:47] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:55:17] PROBLEM - Puppet freshness on mw55 is CRITICAL: Puppet has not run in the last 10 hours [08:00:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.035 seconds [08:28:17] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [08:28:17] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [08:33:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:39:14] PROBLEM - Puppet freshness on cp1028 is CRITICAL: Puppet has not run in the last 10 hours [08:44:20] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.454 seconds [08:51:22] PROBLEM - Apache HTTP on mw35 is CRITICAL: Connection refused [09:17:55] RECOVERY - Apache HTTP on mw35 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.064 second response time [09:19:52] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:29:10] PROBLEM - MySQL Slave Delay on db1007 is CRITICAL: CRIT replication delay 183 seconds [09:30:31] PROBLEM - MySQL Replication Heartbeat on db1007 is CRITICAL: CRIT replication delay 186 seconds [09:32:28] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 9.500 seconds [09:33:04] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 192 seconds [09:34:16] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 233 seconds [09:34:25] PROBLEM - MySQL Slave Delay on db1007 is CRITICAL: CRIT replication delay 183 seconds [09:36:04] RECOVERY - MySQL Slave Delay on db1007 is OK: OK replication delay 3 seconds [09:37:43] RECOVERY - MySQL Replication Heartbeat on db1007 is OK: OK replication delay 0 seconds [09:38:28] RECOVERY - MySQL Replication Heartbeat on db1035 is OK: OK replication delay 0 seconds [09:39:22] RECOVERY - MySQL Slave Delay on db1035 is OK: OK replication delay 0 seconds [09:49:16] PROBLEM - Puppet freshness on db1047 is CRITICAL: Puppet has not run in the last 10 hours [09:49:16] PROBLEM - Puppet freshness on ms-fe1003 is CRITICAL: Puppet has not run in the last 10 hours [09:49:16] PROBLEM - Puppet freshness on ms-fe1004 is CRITICAL: Puppet has not run in the last 10 hours [09:49:16] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [09:49:16] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [09:49:16] PROBLEM - Puppet freshness on sq48 is CRITICAL: Puppet has not run in the last 10 hours [10:05:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:18:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 1.764 seconds [10:48:57] PROBLEM - MySQL Slave Delay on db1007 is CRITICAL: CRIT replication delay 188 seconds [10:48:57] PROBLEM - MySQL Replication Heartbeat on db1007 is CRITICAL: CRIT replication delay 189 seconds [10:53:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:56:09] PROBLEM - Puppet freshness on solr2 is CRITICAL: Puppet has not run in the last 10 hours [10:58:06] PROBLEM - Puppet freshness on vanadium is CRITICAL: Puppet has not run in the last 10 hours [11:03:03] RECOVERY - MySQL Slave Delay on db1007 is OK: OK replication delay 0 seconds [11:03:12] RECOVERY - MySQL Replication Heartbeat on db1007 is OK: OK replication delay 0 seconds [11:03:58] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 5.834 seconds [11:08:09] PROBLEM - Puppet freshness on solr3 is CRITICAL: Puppet has not run in the last 10 hours [11:08:09] PROBLEM - Puppet freshness on solr1003 is CRITICAL: Puppet has not run in the last 10 hours [11:09:03] PROBLEM - Puppet freshness on solr1001 is CRITICAL: Puppet has not run in the last 10 hours [11:39:38] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:52:05] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.132 seconds [11:56:53] PROBLEM - Puppet freshness on brewster is CRITICAL: Puppet has not run in the last 10 hours [12:27:11] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:50] PROBLEM - Puppet freshness on sq81 is CRITICAL: Puppet has not run in the last 10 hours [12:39:38] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.601 seconds [12:51:17] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [12:54:16] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [13:00:16] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours [13:14:22] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:26:49] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.028 seconds [13:37:19] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [13:37:47] New patchset: Faidon; "partman: new generation ceph-ssd recipe" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/41574 [13:38:43] New review: Faidon; "Painfully iterated and tested." [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/41574 [13:38:44] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/41574 [13:43:55] RECOVERY - Host ms-be1004 is UP: PING OK - Packet loss = 0%, RTA = 26.63 ms [13:47:40] PROBLEM - SSH on ms-be1004 is CRITICAL: Connection refused [13:52:55] RECOVERY - SSH on ms-be1004 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [13:58:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:09:34] PROBLEM - NTP on ms-be1004 is CRITICAL: NTP CRITICAL: No response from NTP server [14:09:52] PROBLEM - Host ms-be1003 is DOWN: PING CRITICAL - Packet loss = 100% [14:12:43] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.081 seconds [14:15:43] RECOVERY - Host ms-be1003 is UP: PING OK - Packet loss = 0%, RTA = 26.55 ms [14:19:55] PROBLEM - SSH on ms-be1003 is CRITICAL: Connection refused [14:23:13] PROBLEM - Puppet freshness on stat1 is CRITICAL: Puppet has not run in the last 10 hours [14:23:40] PROBLEM - Host ms-be1003 is DOWN: PING CRITICAL - Packet loss = 100% [14:25:19] RECOVERY - SSH on ms-be1003 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [14:25:28] RECOVERY - Host ms-be1003 is UP: PING OK - Packet loss = 0%, RTA = 26.58 ms [14:29:31] PROBLEM - SSH on ms-be1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:31:10] RECOVERY - SSH on ms-be1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [14:46:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:58:55] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.278 seconds [15:09:16] PROBLEM - Puppet freshness on silver is CRITICAL: Puppet has not run in the last 10 hours [15:09:16] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [15:18:15] New patchset: Alex Monk; "(bug 43517) Change testwiki permissions" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/41579 [15:32:31] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:10] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.788 seconds [16:18:34] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:26:10] Change merged: Demon; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/41579 [16:27:06] !log demon synchronized wmf-config/InitialiseSettings.php 'Deploying I9be1e7ac, testwiki permission changes' [16:27:18] Logged the message, Master [16:31:01] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.026 seconds [16:45:10] PROBLEM - Memcached on virt0 is CRITICAL: Connection refused [16:52:04] PROBLEM - MySQL Slave Delay on db1035 is CRITICAL: CRIT replication delay 189 seconds [16:52:58] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 181 seconds [16:54:46] RECOVERY - MySQL Replication Heartbeat on db1035 is OK: OK replication delay 0 seconds [16:55:31] RECOVERY - MySQL Slave Delay on db1035 is OK: OK replication delay 0 seconds [16:58:22] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [17:04:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:06:19] PROBLEM - Puppet freshness on analytics1007 is CRITICAL: Puppet has not run in the last 10 hours [17:06:19] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: Puppet has not run in the last 10 hours [17:06:19] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: Puppet has not run in the last 10 hours [17:06:19] PROBLEM - Puppet freshness on ms-be1007 is CRITICAL: Puppet has not run in the last 10 hours [17:17:07] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 2.125 seconds [17:36:19] RECOVERY - Memcached on virt0 is OK: TCP OK - 0.002 second response time on port 11000 [17:51:10] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:56:16] PROBLEM - Puppet freshness on mw55 is CRITICAL: Puppet has not run in the last 10 hours [18:01:49] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 5.023 seconds [18:29:30] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [18:29:30] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [18:37:54] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:40:36] PROBLEM - Puppet freshness on cp1028 is CRITICAL: Puppet has not run in the last 10 hours [18:48:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.744 seconds [19:11:39] mo [19:24:07] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:34:18] RECOVERY - Memcached on mc1012 is OK: TCP OK - 0.027 second response time on port 11211 [19:38:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.402 seconds [19:40:42] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/41194 [19:50:30] PROBLEM - Puppet freshness on db1047 is CRITICAL: Puppet has not run in the last 10 hours [19:50:30] PROBLEM - Puppet freshness on sq48 is CRITICAL: Puppet has not run in the last 10 hours [19:50:31] PROBLEM - Puppet freshness on ms-be1010 is CRITICAL: Puppet has not run in the last 10 hours [19:50:31] PROBLEM - Puppet freshness on ms-fe1004 is CRITICAL: Puppet has not run in the last 10 hours [19:50:31] PROBLEM - Puppet freshness on ms-fe1003 is CRITICAL: Puppet has not run in the last 10 hours [19:50:31] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [19:51:40] New patchset: Asher; "missing class parameter" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/41593 [19:51:58] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/41593 [19:57:35] New patchset: Asher; "fixing incorrect syntax in merged change Id4362fdc, jenkins failed to -1" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/41594 [19:58:05] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/41594 [20:03:10] New patchset: Asher; "dbtree: cache ganglia xml data every minute" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/41596 [20:04:34] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/41596 [20:11:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:12:14] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/40554 [20:22:26] binasher: :D :D :D :D [20:22:34] thank you! [20:22:46] do you know when it'll go live? [20:24:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.020 seconds [20:27:41] ori-l: i was just looking at the puppet manifest, and unfortunately every frontend varnish instance will need a full manual restart. the service only subscribes to the package version, not the /etc/default file. i think i'll wait til 1/2 to do that, after puppet has updated the file everywhere [20:28:00] PROBLEM - MySQL Replication Heartbeat on db1007 is CRITICAL: CRIT replication delay 206 seconds [20:28:09] PROBLEM - MySQL Slave Delay on db1007 is CRITICAL: CRIT replication delay 211 seconds [20:28:30] binasher: sounds good to me [20:29:21] if i don't hear from you, is it ok if i check in with you on 1/3? [20:30:17] ori-l: please do, i might need the reminder :) [20:30:44] will do. thanks again for your help with this. [20:32:57] i did update and restart cp1044 just to make sure it's ok, and it is [20:34:27] sweet [20:34:54] RECOVERY - MySQL Replication Heartbeat on db1007 is OK: OK replication delay 0 seconds [20:35:21] RECOVERY - MySQL Slave Delay on db1007 is OK: OK replication delay 0 seconds [20:57:33] PROBLEM - Puppet freshness on solr2 is CRITICAL: Puppet has not run in the last 10 hours [20:57:51] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:42] notpeter: were the Lucene restarts on December 13 on behalf of Patrick? http://wikitech.wikimedia.org/view/Server_admin_log#December_13 [20:59:30] PROBLEM - Puppet freshness on vanadium is CRITICAL: Puppet has not run in the last 10 hours [21:00:28] * robla is trying to figure out when the stuff that Patrick did on Tim's behalf went out [21:02:33] so....I'm guessing LeslieCarr isn't the on call person today, since she's taking the day off [21:02:56] the Search boxes might need to be kicked [21:04:10] https://bugzilla.wikimedia.org/show_bug.cgi?id=42423  Wikimedia wiki search is broken (outputting inconsistent results) [21:05:08] also, http://wikitech.wikimedia.org/view/Server_admin_log from yesterday: "09:05 Nemo_bis: Search reported broken with no results at all returned on en.wikt, (en|ru).source etc. "Lucene on search14 is CRITICAL" since 3h ago." [21:06:19] btw MaxSem noted that search14 is the index for en.wiki so the two things are supposedly unrelated [21:06:24] iirc [21:09:33] PROBLEM - Puppet freshness on solr3 is CRITICAL: Puppet has not run in the last 10 hours [21:09:33] PROBLEM - Puppet freshness on solr1003 is CRITICAL: Puppet has not run in the last 10 hours [21:10:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.021 seconds [21:10:36] PROBLEM - Puppet freshness on solr1001 is CRITICAL: Puppet has not run in the last 10 hours [21:10:52] search14 is fine now [21:11:07] and commons fulltext search is returning results quickly [21:11:27] it probably sucks during index sync [21:15:59] thanks binasher. I'm having a hard time reproducing any problems, actually. I was responding to a ping from andre__ on the subject [21:16:57] yeah, we had the problem of search not working reliably mentioned a few weeks ago in bug 42423, and it came up again yesterday [21:17:23] robla: that bugzilla ticket should probably be closed unless it's worth have a ticket to report new search problems as they happen. search was broken over the thanksgiving holiday, which was when mzmcbride opened it [21:17:38] Which bug? [21:17:49] https://bugzilla.wikimedia.org/show_bug.cgi?id=42423 [21:18:00] sumanah could reproduce it yesterday and somebody could reproduce it two hours ago [21:18:12] though recent issues were about search on mediawiki.org, old issue was about commons [21:18:44] comment 17 might be worth acting on from the mediawiki side [21:18:45] wfm [21:18:46] Just out of curiosity, do search results go in the HTML Squid cache? Does anyone know? [21:19:02] They probably wouldn't, right... [21:19:32] I've become more wary of getting up-to-date data from en.wikipedia.org, heh. [21:19:41] binasher: interesting point. someone want to file comment 17 as a separate bug? [21:19:42] binasher: hence not sure why to close the bugzilla ticket if it could still be reproduced a few hours ago, but maybe I miseed something in the backlog here [21:20:13] Susan: Special:Search does not get squid cached [21:20:23] Okay, right. [21:20:25] I think I knew that. [21:20:31] <^demon> Special pages aren't squid cached generally, iirc. [21:20:38] Most Special pages are booted from the cache, except for the ones that use their own cache system. [21:20:44] it's possible that api.php?opensearch queries do get api cached [21:20:45] querycache [21:21:08] andre__: if search is broken *right now*, then that's a relatively urgent thing that might mean someone needs to kick a box somewhere [21:21:09] I'm not able to reproduce that bug any longer. Though the larger issue seemed to be search support. [21:21:10] but that's mostly just relevant to type-ahead suggestions which are based only on page title match [21:21:12] Has a Lucene person been found? [21:21:42] I saw an RFP or something. [21:21:48] andre__: if, on the other hand, it's general malaise about search, that's something we have two job postings for, as well as something Tim is more active on now [21:21:48] we've interviewed a few candidates but not yet [21:21:52] Lame. [21:22:01] Anyway, that seems like the real resolution to that bug. ;-) [21:22:13] definitely [21:22:15] ah, nice [21:22:27] rainman-sr served us well. But all good things pass. [21:23:08] so....I guess I'll file comment 17 as a bug [21:23:50] robla: i think that should go against the MWSearch extension [21:24:59] You can probably mark it "easy". [21:25:33] Though I'm not sure how easy it actually is to distinguish 0 results due to 0 results or 0 rseults due to a broken search box. [21:27:00] results * [21:27:47] the should either be a 10s timeout on the lucene query getting hit, or a non-ok return code [21:28:15] Susan: no...."easy" is for stuff that is easy for someone new to the codebase [21:28:30] robla: I'm familiar with the keyword. [21:28:49] I guess anything search-related isn't easy these days, though. [21:28:56] Susan: then why do you keep marking/suggesting non-easy bugs as "easy"? :-P [21:29:05] Heh. [21:29:18] Someone suggested I was doing it as a taunt. [21:29:27] robla: i think everyone actively taking eng tickets is new to MWSearch :) [21:29:39] That it would agitate a developer to fixing the bug sooner, because it was marked easy. [21:29:57] Though really I do it because I think the bug is easy, and bugs marked as easy seem to get attention sooner. [21:30:13] but this should be an easy task for getting somewhat acquainted to it [21:33:23] binasher: do you know if it's possible to tell outside of Lucene whether zero results is due to Lucene indexing failure or due to there actually being zero results? [21:43:54] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:49:22] New review: Ori.livneh; "Because the current EventLogging architecture makes it easy to subscribe to the event stream and stu..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/41206 [21:50:15] could i bug someone to merge? https://gerrit.wikimedia.org/r/#/c/41206/ & https://gerrit.wikimedia.org/r/#/c/41204/ (both are one- or two-line changes) [21:56:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.522 seconds [21:58:16] robla: a lot of the erroneous 0 result responses are coming from query timeouts, that case should be straight forward to catch [21:58:36] PROBLEM - Puppet freshness on brewster is CRITICAL: Puppet has not run in the last 10 hours [21:58:52] yup, makes sense [22:00:04] I've asked Tim to look at it first. https://bugzilla.wikimedia.org/show_bug.cgi?id=43544 [22:00:39] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/41204 [22:01:15] it may also be helpful to have some application-level monitoring if we don't have any yet (e.g. actually query Lucene for something that should return >x results, where x is a reasonably large number) [22:02:11] New review: Asher; "http://mcfunley.com/why-mongodb-never-worked-out-at-etsy" [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/41206 [22:02:13] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/41206 [22:02:42] binasher: thanks :) [22:03:00] i forwarded that blog post to the e3 list! [22:03:18] oh! hehe [22:03:22] because people had been bugging me about various forms NewHotnessDB [22:03:44] i also enjoyed http://lucumr.pocoo.org/2012/12/29/sql-is-agile/ [22:04:04] from mitsuhiko / armin ronacher (the guy who wrote flask, jinja2, etc.) [22:06:06] binasher: lots of 'Error connecting to 10.0.6.73: User 'wikiadmin' has exceeded the 'max_user_connections' resource (current value: 80)' [22:06:16] there are 16 boxes with 12 processes each right? [22:06:29] yep [22:06:44] so that limit is kind of low [22:06:52] yep [22:07:23] i don't want 16*20 job runners working on enwiki at once [22:08:12] but a way to manage max concurrent jobs per wiki that doesn't involve db limits would be welcome [22:08:48] * 16*12 [22:20:29] ori-l: i think projects either map best to a column oriented db (aka analytics putting structured log data in cassandra if they deploy it) or they should use mysql, if for no other reasons than what's argued in those two blog posts. for the first case, we should probably select and standardize on one db [22:22:34] binasher: yes, i'm strongly committed to mysql; the change set is mostly to satisfy my personal curiosity [22:23:19] New patchset: Dereckson; "(bug 40879) Enable Collection on ba.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/41809 [22:23:43] there's a slight impedance mismatch when converting from JSON event data to SQL, which is really a variant of the standard object-relational impedence mismatch [22:23:52] i.e., how do you model nested objects, things like that [22:24:24] but because we use JSON schema to specify constraints on the structure of events, it's been possible for us to identify a subset that can translates cleanly [22:28:09] and with that in place, it's kind of a no-brainer: putting the data in mysql means being able to easily join it with production data, not having to enforce constraints by hand, plus the ability for analysts to use their SQL knowledge for working with the data [22:28:57] at some point, there will probably be a mongodb vs. cassandra throwdown [22:29:28] can i watch? [22:31:22] i think people often don't realize that a lot of NoSQL simply means offloading work from computers onto people (relational query planner -> writing mapreduce jobs, guaranteed constraints vs. having to write a ton of error-handling code to deal w/inconsistencies, etc.) [22:31:25] anyways, [22:31:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:33:06] it may be a rant but it's true [22:35:23] AaronSchulz: i unfortunately realized all this after spending two months writing all data into redis and then showing it to analysts who were a) uniformly very impressed, and b) never went on to do anything with it [22:36:01] i looked around for a document store with a strongly relational model but it seems like the current crop is strongly focussed on making CAP tradeoffs to scale [22:38:45] it's a rant that more people need to hear :) [22:39:33] PROBLEM - Puppet freshness on sq81 is CRITICAL: Puppet has not run in the last 10 hours [22:39:52] binasher: http://browsertoolkit.com/fault-tolerance.png [22:40:36] love that! [22:42:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 1.602 seconds [22:47:57] i don't want 16*20 job runners working on enwiki at once # I'm not sure I understand why. That's too many? [22:52:36] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [22:55:14] Susan: some job types are brokenly resource intense and having a ton of them running at once was impacting the site. i.e. jobs calling BacklinkCache::getLinks running select with order by on 10+ mil row sets with no limit. the limit on concurrent job execution is a symptom of jobs behaving badly. [22:55:36] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [22:55:56] Hmm, right. [22:56:24] Is there better job queue monitoring these days? [22:56:35] I remember the biggest issue used to be even figuring out what the oldest job was. [22:56:48] Or even getting an accurate count of the jobs in the queue... [22:57:08] New patchset: Dereckson; "(bug 43532) Enable PostEdit on ml.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/41812 [22:57:31] 7. [22:59:02] https://en.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=statistics&format=jsonfm [22:59:22] Seems it's around 200K right now. [22:59:39] I was close. [23:00:25] it's better than it was, but not good enough to point out jobs behaving badly [23:01:31] https://bugzilla.wikimedia.org/show_bug.cgi?id=9518 [23:01:36] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours [23:01:39] the need to limit concurrency is unfortunate, especially since the change to use refreshLinks instead of just refreshLinks2 greatly increased the number of jobs that get created [23:01:49] "Woefully" is a Rob Church word to use, heh. [23:02:57] we don't use the data behind Special:Statistics internally [23:03:11] It was removed from Special:Statistics. [23:03:16] What's used internally? [23:03:43] select count(*) from job :) [23:03:48] Heh. [23:06:03] siteinfo statistics are from innodb table stats which are just rough inaccurate estimates, pulled from a random slave [23:06:12] Right. [23:07:11] So I guess there's a job_id column that be used to approximately job age. Still no timestamp field, though? [23:07:20] Hm. [23:07:43] the json should probably include "this is all a wild guess": 1 [23:07:48] Susan: That sentence makes no sense. [23:07:56] Though I did include most of the needed words. [23:08:04] Heh. [23:08:29] I'm not sure whether the value still makes sense in the API. [23:08:39] The use-cases are still unclear. Much as they were in the GUI. [23:09:29] mysql:wikiadmin@db63 [enwiki]> select min(job_timestamp) from job where job_attempts < 3; [23:09:30] +--------------------+ [23:09:31] | min(job_timestamp) | [23:09:34] +--------------------+ [23:09:34] | 20121213101221 | [23:09:35] +--------------------+ [23:09:36] 1 row in set (0.12 sec) [23:09:43] Hmmm. [23:10:04] https://www.mediawiki.org/wiki/Manual:Job_table So it is. [23:10:08] I should learn to read. [23:10:30] the very old stuff currently in enwiki.job are webVideoTranscode jobs [23:10:39] AaronSchulz: is webVideoTranscode broken? [23:11:14] 20121213 is much less happy than 20121231. [23:12:00] binasher: what does job_attempts say? [23:13:13] AaronSchulz: 1 [23:13:19] mysql:wikiadmin@db63 [enwiki]> select min(job_timestamp) from job where job_attempts = 0 and job_cmd != "webVideoTranscode"; [23:13:19] +--------------------+ [23:13:21] | min(job_timestamp) | [23:13:22] +--------------------+ [23:13:23] | 20121231221020 | [23:13:24] +--------------------+ [23:13:46] the oldest job that hasn't been attempted yet is about an hour old [23:14:33] New patchset: Dereckson; "(bug 42721) tr.wikisource has been renamed into Vikikaynak" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/41813 [23:14:35] I hadn't realized job_timestamp and job_attempts had been added. So there's good progress being made. That's nice. :-) [23:15:26] AaronSchulz: is JobQueueDB::claimRandom generally grabbing the last or the first now, or is it fairly random? i forget where we left that [23:15:42] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:17:56] https://bugzilla.wikimedia.org/show_bug.cgi?id=42614 ? [23:18:31] binasher: it's last and first [23:19:52] https://gerrit.wikimedia.org/r/#/c/38466/4 is still pending, though Tim and I had doubts about that and the current state [23:20:19] AaronSchulz: i liked your idea of just pulling the first with a random offset.. if number of jobs mapped to a memcached counter, it could be ok [23:22:05] AaronSchulz: i disagree with tim's last comment on the change [23:22:59] yeah, it doesn't need an order, mysql will pick an index [23:23:05] in this case the cmd_token_id one [23:23:30] though I wonder what other rdbms will do, since it is not well defined in the strict sense [23:29:57] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.028 seconds [23:38:30] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [23:44:17] robla: yep, those were to push out a new version of lucene [23:44:31] robla: well, the restarts were because they were acting funky [23:45:41] * robla smiles at the thought of a search engine acting funky :-) [23:45:57] they were in dire need of a shower, I tell you [23:46:28] 24 results, fo' sho' [23:46:34] hahaha [23:48:36] thanks for the update. I think, after asking you about that, we established that things are (as of this instant) in ok shape, but something we could use a little more monitoring of