[00:19:23] PROBLEM - Puppet freshness on mw1102 is CRITICAL: Puppet has not run in the last 10 hours [00:19:24] PROBLEM - Puppet freshness on mw1158 is CRITICAL: Puppet has not run in the last 10 hours [00:23:20] PROBLEM - Puppet freshness on mw1118 is CRITICAL: Puppet has not run in the last 10 hours [00:29:23] PROBLEM - Puppet freshness on mw1085 is CRITICAL: Puppet has not run in the last 10 hours [00:31:33] PROBLEM - Puppet freshness on cp1034 is CRITICAL: Puppet has not run in the last 10 hours [00:46:19] PROBLEM - Puppet freshness on mw35 is CRITICAL: Puppet has not run in the last 10 hours [00:52:36] Susan: where else would it be? [00:52:48] Susan: the most obvious spot for a user is their user page [00:53:04] it just delegates to a special page, anyway [00:53:22] there's also a discovery endpoint that can be used [02:03:47] New review: AzaToth; "It's not when the draft is made or updated, but when the draft is published. i.e. when it's released..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50044 [02:15:20] PROBLEM - Puppet freshness on mw44 is CRITICAL: Puppet has not run in the last 10 hours [02:17:18] PROBLEM - Puppet freshness on mw1092 is CRITICAL: Puppet has not run in the last 10 hours [02:18:18] PROBLEM - Puppet freshness on colby is CRITICAL: Puppet has not run in the last 10 hours [02:23:18] PROBLEM - Puppet freshness on mw62 is CRITICAL: Puppet has not run in the last 10 hours [02:58:29] PROBLEM - Host mw27 is DOWN: PING CRITICAL - Packet loss = 100% [02:59:58] RECOVERY - Host mw27 is UP: PING OK - Packet loss = 0%, RTA = 26.57 ms [03:02:50] PROBLEM - Apache HTTP on mw27 is CRITICAL: Connection refused [03:05:08] PROBLEM - Host mw27 is DOWN: PING CRITICAL - Packet loss = 100% [03:05:48] RECOVERY - Host mw27 is UP: PING OK - Packet loss = 0%, RTA = 26.55 ms [03:51:18] PROBLEM - Puppet freshness on db67 is CRITICAL: Puppet has not run in the last 10 hours [03:52:18] PROBLEM - Puppet freshness on ms-fe2 is CRITICAL: Puppet has not run in the last 10 hours [04:13:18] PROBLEM - Puppet freshness on mw1126 is CRITICAL: Puppet has not run in the last 10 hours [04:14:20] PROBLEM - Puppet freshness on mw1099 is CRITICAL: Puppet has not run in the last 10 hours [04:16:18] PROBLEM - Puppet freshness on mw1152 is CRITICAL: Puppet has not run in the last 10 hours [04:16:18] PROBLEM - Puppet freshness on mw69 is CRITICAL: Puppet has not run in the last 10 hours [04:17:19] PROBLEM - Puppet freshness on mw1001 is CRITICAL: Puppet has not run in the last 10 hours [04:17:19] PROBLEM - Puppet freshness on mw1103 is CRITICAL: Puppet has not run in the last 10 hours [04:18:18] PROBLEM - Puppet freshness on mw1025 is CRITICAL: Puppet has not run in the last 10 hours [04:18:18] PROBLEM - Puppet freshness on mw1109 is CRITICAL: Puppet has not run in the last 10 hours [04:19:18] PROBLEM - Puppet freshness on grosley is CRITICAL: Puppet has not run in the last 10 hours [04:19:18] PROBLEM - Puppet freshness on locke is CRITICAL: Puppet has not run in the last 10 hours [04:19:18] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [04:19:18] PROBLEM - Puppet freshness on mw1155 is CRITICAL: Puppet has not run in the last 10 hours [04:19:18] PROBLEM - Puppet freshness on mw1116 is CRITICAL: Puppet has not run in the last 10 hours [04:19:19] PROBLEM - Puppet freshness on tola is CRITICAL: Puppet has not run in the last 10 hours [04:21:18] PROBLEM - Puppet freshness on hooft is CRITICAL: Puppet has not run in the last 10 hours [04:21:18] PROBLEM - Puppet freshness on ms6 is CRITICAL: Puppet has not run in the last 10 hours [04:21:18] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [04:22:20] PROBLEM - Puppet freshness on mw1012 is CRITICAL: Puppet has not run in the last 10 hours [04:22:20] PROBLEM - Puppet freshness on mw48 is CRITICAL: Puppet has not run in the last 10 hours [04:23:21] PROBLEM - Puppet freshness on mw1063 is CRITICAL: Puppet has not run in the last 10 hours [04:25:43] run, puppet, run! [04:31:37] root@mw1063:~# ps -C puppet -o stime,args [04:31:37] STIME COMMAND [04:31:38] Mar22 /usr/bin/ruby1.8 /usr/bin/puppet agent --onetime --verbose --no-daemonize --no-splay --show_diff [04:32:30] and you know what its parent is? timeout 1800 [04:33:41] it's process management as usual, except waiting on a defunct sh instead of the usual apt-get [04:35:18] PROBLEM - Puppet freshness on cp3022 is CRITICAL: Puppet has not run in the last 10 hours [04:50:00] PROBLEM - NTP on cp1023 is CRITICAL: NTP CRITICAL: Offset unknown [04:56:01] RECOVERY - NTP on cp1023 is OK: NTP OK: Offset 0.06014597416 secs [05:04:38] Ryan_Lane: Changing the HTTP response code for only certain pages seems really nasty to me. [05:08:20] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [05:13:08] PROBLEM - Packetloss_Average on gadolinium is CRITICAL: CRITICAL: packet_loss_average is 99.0 (gt 8.0) [05:17:17] RECOVERY - Packetloss_Average on gadolinium is OK: OK: packet_loss_average is -3.33333 [05:18:46] What needs Ruby on mw1063? [05:45:05] https://bugzilla.wikimedia.org/show_bug.cgi?id=46528 [05:45:16] Does anyone have thoughts about enabling the RSS extension on all Wikimedia wikis? [06:13:17] PROBLEM - Puppet freshness on mw1077 is CRITICAL: Puppet has not run in the last 10 hours [06:15:17] PROBLEM - Puppet freshness on mw1104 is CRITICAL: Puppet has not run in the last 10 hours [06:18:17] PROBLEM - Puppet freshness on mw1043 is CRITICAL: Puppet has not run in the last 10 hours [06:19:17] PROBLEM - Puppet freshness on mw1080 is CRITICAL: Puppet has not run in the last 10 hours [06:21:17] PROBLEM - Puppet freshness on mw1003 is CRITICAL: Puppet has not run in the last 10 hours [06:24:17] PROBLEM - Puppet freshness on virt5 is CRITICAL: Puppet has not run in the last 10 hours [06:25:20] PROBLEM - Puppet freshness on mw1089 is CRITICAL: Puppet has not run in the last 10 hours [06:26:17] PROBLEM - Puppet freshness on mw1129 is CRITICAL: Puppet has not run in the last 10 hours [06:29:17] PROBLEM - Puppet freshness on mw1141 is CRITICAL: Puppet has not run in the last 10 hours [06:33:17] PROBLEM - Puppet freshness on gadolinium is CRITICAL: Puppet has not run in the last 10 hours [06:37:17] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: Puppet has not run in the last 10 hours [06:37:18] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [06:37:18] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: Puppet has not run in the last 10 hours [06:37:18] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [06:37:18] PROBLEM - Puppet freshness on analytics1002 is CRITICAL: Puppet has not run in the last 10 hours [06:49:05] New review: Tim Starling; "It seems pretty harmless to me. We already have several packages installed everywhere for the conven..." [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/50306 [07:16:13] New review: Ori.livneh; "Oh, my mistake. This is fairly innocuous, then." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/50044 [07:16:27] RECOVERY - Puppet freshness on cp3010 is OK: puppet ran at Mon Mar 25 07:16:18 UTC 2013 [07:34:18] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [07:34:18] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [07:34:18] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours [07:34:18] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [07:47:32] New review: Ori.livneh; "I can't think of a reason why this would matter, but if you're stuck, try referring to the class as ..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54970 [08:14:19] PROBLEM - Puppet freshness on analytics1010 is CRITICAL: Puppet has not run in the last 10 hours [08:14:19] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: Puppet has not run in the last 10 hours [08:14:19] PROBLEM - Puppet freshness on cerium is CRITICAL: Puppet has not run in the last 10 hours [08:14:19] PROBLEM - Puppet freshness on cp1028 is CRITICAL: Puppet has not run in the last 10 hours [08:14:19] PROBLEM - Puppet freshness on cp3003 is CRITICAL: Puppet has not run in the last 10 hours [08:15:27] ... [08:17:18] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [08:19:45] Hi Bsadowski1. [08:22:17] PROBLEM - Puppet freshness on mw1093 is CRITICAL: Puppet has not run in the last 10 hours [09:19:48] New patchset: Mark Bergsma; "Move replacement of Cache-Control header from backend to frontend" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55548 [09:20:57] PROBLEM - SSH on amslvs1 is CRITICAL: Server answer: [09:21:57] RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [09:24:42] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55548 [09:42:18] PROBLEM - Puppet freshness on constable is CRITICAL: Puppet has not run in the last 10 hours [09:46:17] PROBLEM - Puppet freshness on gallium is CRITICAL: Puppet has not run in the last 10 hours [09:59:53] New patchset: Mark Bergsma; "Cap the cache TTL to 300s on mobile frontends" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55551 [10:01:05] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55551 [10:03:47] New patchset: Mark Bergsma; "ttl needs explicit time spec" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55552 [10:04:20] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55552 [10:11:22] New patchset: Mark Bergsma; "Use pass in vcl_recv for test.* instead of hit_for_pass in vcl_fetch" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55555 [10:12:26] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55555 [10:20:18] PROBLEM - Puppet freshness on mw1102 is CRITICAL: Puppet has not run in the last 10 hours [10:20:18] PROBLEM - Puppet freshness on mw1158 is CRITICAL: Puppet has not run in the last 10 hours [10:23:37] New patchset: Mark Bergsma; "Let Varnish logic set TTL on 4xx, but cap at a configured TTL" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55556 [10:24:17] PROBLEM - Puppet freshness on mw1118 is CRITICAL: Puppet has not run in the last 10 hours [10:30:17] PROBLEM - Puppet freshness on mw1085 is CRITICAL: Puppet has not run in the last 10 hours [10:32:18] PROBLEM - Puppet freshness on cp1034 is CRITICAL: Puppet has not run in the last 10 hours [10:34:04] New review: Hashar; "(2 comments)" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/54970 [10:47:19] PROBLEM - Puppet freshness on mw35 is CRITICAL: Puppet has not run in the last 10 hours [10:59:15] New patchset: Mark Bergsma; "Let Varnish logic set TTL on 4xx, but cap at a configured TTL" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55556 [11:00:19] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55556 [11:19:36] New patchset: Mark Bergsma; "Use tabs as field separators for event logging" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55558 [11:20:43] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55558 [11:22:19] RECOVERY - Puppet freshness on arsenic is OK: puppet ran at Mon Mar 25 11:22:01 UTC 2013 [11:24:56] New patchset: Mark Bergsma; "Formatting, unbreak the puppet cron job" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55559 [11:25:40] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55559 [11:29:47] RECOVERY - Puppet freshness on niobium is OK: puppet ran at Mon Mar 25 11:29:46 UTC 2013 [11:31:37] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Mon Mar 25 11:31:34 UTC 2013 [11:33:59] RECOVERY - Puppet freshness on strontium is OK: puppet ran at Mon Mar 25 11:33:50 UTC 2013 [11:37:27] RECOVERY - Puppet freshness on cp3019 is OK: puppet ran at Mon Mar 25 11:37:25 UTC 2013 [11:40:47] RECOVERY - Puppet freshness on cp3020 is OK: puppet ran at Mon Mar 25 11:40:39 UTC 2013 [11:44:19] RECOVERY - Puppet freshness on tarin is OK: puppet ran at Mon Mar 25 11:44:13 UTC 2013 [11:45:00] RECOVERY - Puppet freshness on calcium is OK: puppet ran at Mon Mar 25 11:44:50 UTC 2013 [11:45:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:46:08] RECOVERY - Puppet freshness on virt3 is OK: puppet ran at Mon Mar 25 11:45:57 UTC 2013 [11:47:07] RECOVERY - Puppet freshness on mc6 is OK: puppet ran at Mon Mar 25 11:47:03 UTC 2013 [11:47:37] RECOVERY - Puppet freshness on wtp1 is OK: puppet ran at Mon Mar 25 11:47:28 UTC 2013 [11:48:37] RECOVERY - Puppet freshness on harmon is OK: puppet ran at Mon Mar 25 11:48:36 UTC 2013 [11:48:57] RECOVERY - Puppet freshness on barium is OK: puppet ran at Mon Mar 25 11:48:51 UTC 2013 [11:49:09] RECOVERY - Puppet freshness on bast1001 is OK: puppet ran at Mon Mar 25 11:48:56 UTC 2013 [11:49:09] RECOVERY - Puppet freshness on mc1013 is OK: puppet ran at Mon Mar 25 11:49:02 UTC 2013 [11:49:58] RECOVERY - Puppet freshness on db1029 is OK: puppet ran at Mon Mar 25 11:49:53 UTC 2013 [11:50:27] RECOVERY - Puppet freshness on mc13 is OK: puppet ran at Mon Mar 25 11:50:19 UTC 2013 [11:50:38] RECOVERY - Puppet freshness on db71 is OK: puppet ran at Mon Mar 25 11:50:32 UTC 2013 [11:50:48] RECOVERY - Puppet freshness on ms-fe1002 is OK: puppet ran at Mon Mar 25 11:50:38 UTC 2013 [11:50:48] RECOVERY - Puppet freshness on db1031 is OK: puppet ran at Mon Mar 25 11:50:38 UTC 2013 [11:50:48] RECOVERY - Puppet freshness on ms-be1003 is OK: puppet ran at Mon Mar 25 11:50:38 UTC 2013 [11:50:58] RECOVERY - Puppet freshness on solr1001 is OK: puppet ran at Mon Mar 25 11:50:55 UTC 2013 [11:52:08] RECOVERY - Puppet freshness on lvs2 is OK: puppet ran at Mon Mar 25 11:52:00 UTC 2013 [11:52:18] RECOVERY - Puppet freshness on lardner is OK: puppet ran at Mon Mar 25 11:52:10 UTC 2013 [11:52:27] RECOVERY - Puppet freshness on db69 is OK: puppet ran at Mon Mar 25 11:52:20 UTC 2013 [11:52:59] RECOVERY - Puppet freshness on titanium is OK: puppet ran at Mon Mar 25 11:52:45 UTC 2013 [11:53:39] RECOVERY - Puppet freshness on db1036 is OK: puppet ran at Mon Mar 25 11:53:27 UTC 2013 [11:53:58] RECOVERY - Puppet freshness on ssl1003 is OK: puppet ran at Mon Mar 25 11:53:53 UTC 2013 [11:54:07] RECOVERY - Puppet freshness on mw123 is OK: puppet ran at Mon Mar 25 11:53:58 UTC 2013 [11:54:08] RECOVERY - Puppet freshness on mw106 is OK: puppet ran at Mon Mar 25 11:54:03 UTC 2013 [11:54:18] RECOVERY - Puppet freshness on ssl1001 is OK: puppet ran at Mon Mar 25 11:54:13 UTC 2013 [11:54:27] RECOVERY - Puppet freshness on db39 is OK: puppet ran at Mon Mar 25 11:54:19 UTC 2013 [11:54:27] RECOVERY - Puppet freshness on ersch is OK: puppet ran at Mon Mar 25 11:54:19 UTC 2013 [11:54:27] RECOVERY - Puppet freshness on solr1002 is OK: puppet ran at Mon Mar 25 11:54:24 UTC 2013 [11:54:47] RECOVERY - Puppet freshness on db78 is OK: puppet ran at Mon Mar 25 11:54:41 UTC 2013 [11:54:47] RECOVERY - Puppet freshness on virt2 is OK: puppet ran at Mon Mar 25 11:54:41 UTC 2013 [11:54:47] RECOVERY - Puppet freshness on virt4 is OK: puppet ran at Mon Mar 25 11:54:41 UTC 2013 [11:54:48] RECOVERY - Puppet freshness on lvs4 is OK: puppet ran at Mon Mar 25 11:54:41 UTC 2013 [11:54:48] RECOVERY - Puppet freshness on caesium is OK: puppet ran at Mon Mar 25 11:54:41 UTC 2013 [11:54:48] RECOVERY - Puppet freshness on ssl1002 is OK: puppet ran at Mon Mar 25 11:54:41 UTC 2013 [11:54:57] RECOVERY - Puppet freshness on mw1199 is OK: puppet ran at Mon Mar 25 11:54:47 UTC 2013 [11:54:57] RECOVERY - Puppet freshness on wtp1001 is OK: puppet ran at Mon Mar 25 11:54:47 UTC 2013 [11:54:57] RECOVERY - Puppet freshness on ssl1004 is OK: puppet ran at Mon Mar 25 11:54:52 UTC 2013 [11:54:57] RECOVERY - Puppet freshness on cerium is OK: puppet ran at Mon Mar 25 11:54:52 UTC 2013 [11:54:57] RECOVERY - Puppet freshness on search35 is OK: puppet ran at Mon Mar 25 11:54:52 UTC 2013 [11:54:57] RECOVERY - Puppet freshness on mc1011 is OK: puppet ran at Mon Mar 25 11:54:52 UTC 2013 [11:54:58] RECOVERY - Puppet freshness on lvs1 is OK: puppet ran at Mon Mar 25 11:54:52 UTC 2013 [11:54:58] RECOVERY - Puppet freshness on mc1016 is OK: puppet ran at Mon Mar 25 11:54:52 UTC 2013 [11:55:07] RECOVERY - Puppet freshness on mc5 is OK: puppet ran at Mon Mar 25 11:54:57 UTC 2013 [11:55:07] RECOVERY - Puppet freshness on mc14 is OK: puppet ran at Mon Mar 25 11:54:57 UTC 2013 [11:55:07] RECOVERY - Puppet freshness on mw1069 is OK: puppet ran at Mon Mar 25 11:55:02 UTC 2013 [11:55:19] RECOVERY - Puppet freshness on db1032 is OK: puppet ran at Mon Mar 25 11:55:07 UTC 2013 [11:55:19] RECOVERY - Puppet freshness on mc1015 is OK: puppet ran at Mon Mar 25 11:55:07 UTC 2013 [11:55:19] RECOVERY - Puppet freshness on mc9 is OK: puppet ran at Mon Mar 25 11:55:07 UTC 2013 [11:55:19] RECOVERY - Puppet freshness on mc8 is OK: puppet ran at Mon Mar 25 11:55:07 UTC 2013 [11:55:19] RECOVERY - Puppet freshness on mc12 is OK: puppet ran at Mon Mar 25 11:55:07 UTC 2013 [11:55:19] RECOVERY - Puppet freshness on ssl3 is OK: puppet ran at Mon Mar 25 11:55:07 UTC 2013 [11:55:19] RECOVERY - Puppet freshness on mc1012 is OK: puppet ran at Mon Mar 25 11:55:12 UTC 2013 [11:55:19] RECOVERY - Puppet freshness on db1047 is OK: puppet ran at Mon Mar 25 11:55:12 UTC 2013 [11:55:19] RECOVERY - Puppet freshness on db66 is OK: puppet ran at Mon Mar 25 11:55:12 UTC 2013 [11:55:27] RECOVERY - Puppet freshness on db1013 is OK: puppet ran at Mon Mar 25 11:55:17 UTC 2013 [11:55:27] RECOVERY - Puppet freshness on xenon is OK: puppet ran at Mon Mar 25 11:55:17 UTC 2013 [11:55:27] RECOVERY - Puppet freshness on potassium is OK: puppet ran at Mon Mar 25 11:55:17 UTC 2013 [11:55:27] RECOVERY - Puppet freshness on wtp1003 is OK: puppet ran at Mon Mar 25 11:55:22 UTC 2013 [11:55:27] RECOVERY - Puppet freshness on wtp1002 is OK: puppet ran at Mon Mar 25 11:55:23 UTC 2013 [11:55:27] RECOVERY - Puppet freshness on ms-fe1001 is OK: puppet ran at Mon Mar 25 11:55:23 UTC 2013 [11:55:27] RECOVERY - Puppet freshness on nitrogen is OK: puppet ran at Mon Mar 25 11:55:23 UTC 2013 [11:55:39] RECOVERY - Puppet freshness on mc3 is OK: puppet ran at Mon Mar 25 11:55:29 UTC 2013 [11:55:39] RECOVERY - Puppet freshness on zhen is OK: puppet ran at Mon Mar 25 11:55:29 UTC 2013 [11:55:39] RECOVERY - Puppet freshness on wtp1004 is OK: puppet ran at Mon Mar 25 11:55:29 UTC 2013 [11:55:39] RECOVERY - Puppet freshness on snapshot1002 is OK: puppet ran at Mon Mar 25 11:55:29 UTC 2013 [11:55:39] RECOVERY - Puppet freshness on mc10 is OK: puppet ran at Mon Mar 25 11:55:29 UTC 2013 [11:55:39] RECOVERY - Puppet freshness on praseodymium is OK: puppet ran at Mon Mar 25 11:55:34 UTC 2013 [11:55:39] RECOVERY - Puppet freshness on mc15 is OK: puppet ran at Mon Mar 25 11:55:34 UTC 2013 [11:55:40] RECOVERY - Puppet freshness on solr3 is OK: puppet ran at Mon Mar 25 11:55:34 UTC 2013 [11:55:40] RECOVERY - Puppet freshness on pc2 is OK: puppet ran at Mon Mar 25 11:55:34 UTC 2013 [11:55:41] RECOVERY - Puppet freshness on manutius is OK: puppet ran at Mon Mar 25 11:55:34 UTC 2013 [11:55:47] RECOVERY - Puppet freshness on pc1002 is OK: puppet ran at Mon Mar 25 11:55:39 UTC 2013 [11:55:47] RECOVERY - Puppet freshness on pc3 is OK: puppet ran at Mon Mar 25 11:55:39 UTC 2013 [11:55:47] RECOVERY - Puppet freshness on ms-be12 is OK: puppet ran at Mon Mar 25 11:55:39 UTC 2013 [11:55:47] RECOVERY - Puppet freshness on db62 is OK: puppet ran at Mon Mar 25 11:55:39 UTC 2013 [11:55:47] RECOVERY - Puppet freshness on mw86 is OK: puppet ran at Mon Mar 25 11:55:45 UTC 2013 [11:55:47] RECOVERY - Puppet freshness on mc1010 is OK: puppet ran at Mon Mar 25 11:55:45 UTC 2013 [11:55:47] RECOVERY - Puppet freshness on db1030 is OK: puppet ran at Mon Mar 25 11:55:46 UTC 2013 [11:55:57] RECOVERY - Puppet freshness on db38 is OK: puppet ran at Mon Mar 25 11:55:51 UTC 2013 [11:55:57] RECOVERY - Puppet freshness on mc2 is OK: puppet ran at Mon Mar 25 11:55:51 UTC 2013 [11:55:57] RECOVERY - Puppet freshness on ocg3 is OK: puppet ran at Mon Mar 25 11:55:51 UTC 2013 [11:55:57] RECOVERY - Puppet freshness on ms-be1008 is OK: puppet ran at Mon Mar 25 11:55:51 UTC 2013 [11:55:57] RECOVERY - Puppet freshness on mc4 is OK: puppet ran at Mon Mar 25 11:55:51 UTC 2013 [11:55:57] RECOVERY - Puppet freshness on lvs3 is OK: puppet ran at Mon Mar 25 11:55:51 UTC 2013 [11:55:58] RECOVERY - Puppet freshness on es1010 is OK: puppet ran at Mon Mar 25 11:55:51 UTC 2013 [11:55:58] RECOVERY - Puppet freshness on search26 is OK: puppet ran at Mon Mar 25 11:55:51 UTC 2013 [11:55:59] RECOVERY - Puppet freshness on ms-be5 is OK: puppet ran at Mon Mar 25 11:55:51 UTC 2013 [11:55:59] RECOVERY - Puppet freshness on helium is OK: puppet ran at Mon Mar 25 11:55:51 UTC 2013 [11:56:00] RECOVERY - Puppet freshness on ms-be1001 is OK: puppet ran at Mon Mar 25 11:55:51 UTC 2013 [11:56:00] RECOVERY - Puppet freshness on mw1205 is OK: puppet ran at Mon Mar 25 11:55:51 UTC 2013 [11:56:01] RECOVERY - Puppet freshness on mw1169 is OK: puppet ran at Mon Mar 25 11:55:56 UTC 2013 [11:56:01] RECOVERY - Puppet freshness on db1037 is OK: puppet ran at Mon Mar 25 11:55:56 UTC 2013 [11:56:02] RECOVERY - Puppet freshness on mw108 is OK: puppet ran at Mon Mar 25 11:55:56 UTC 2013 [11:56:02] RECOVERY - Puppet freshness on search31 is OK: puppet ran at Mon Mar 25 11:55:56 UTC 2013 [11:56:03] RECOVERY - Puppet freshness on mw101 is OK: puppet ran at Mon Mar 25 11:55:56 UTC 2013 [11:56:07] RECOVERY - Puppet freshness on solr2 is OK: puppet ran at Mon Mar 25 11:56:02 UTC 2013 [11:56:07] RECOVERY - Puppet freshness on db36 is OK: puppet ran at Mon Mar 25 11:56:02 UTC 2013 [11:56:07] RECOVERY - Puppet freshness on mc7 is OK: puppet ran at Mon Mar 25 11:56:02 UTC 2013 [11:56:17] RECOVERY - Puppet freshness on ms-be11 is OK: puppet ran at Mon Mar 25 11:56:07 UTC 2013 [11:56:17] RECOVERY - Puppet freshness on search29 is OK: puppet ran at Mon Mar 25 11:56:07 UTC 2013 [11:56:17] RECOVERY - Puppet freshness on search32 is OK: puppet ran at Mon Mar 25 11:56:07 UTC 2013 [11:56:17] RECOVERY - Puppet freshness on solr1003 is OK: puppet ran at Mon Mar 25 11:56:07 UTC 2013 [11:56:17] RECOVERY - Puppet freshness on mc11 is OK: puppet ran at Mon Mar 25 11:56:07 UTC 2013 [11:56:17] RECOVERY - Puppet freshness on ms10 is OK: puppet ran at Mon Mar 25 11:56:07 UTC 2013 [11:57:27] RECOVERY - Puppet freshness on ms-be1011 is OK: puppet ran at Mon Mar 25 11:57:17 UTC 2013 [11:57:27] RECOVERY - Puppet freshness on mw1073 is OK: puppet ran at Mon Mar 25 11:57:17 UTC 2013 [11:57:27] RECOVERY - Puppet freshness on mw1011 is OK: puppet ran at Mon Mar 25 11:57:17 UTC 2013 [11:57:27] RECOVERY - Puppet freshness on mw1068 is OK: puppet ran at Mon Mar 25 11:57:22 UTC 2013 [11:57:27] RECOVERY - Puppet freshness on mw1040 is OK: puppet ran at Mon Mar 25 11:57:22 UTC 2013 [11:57:37] RECOVERY - Puppet freshness on mw43 is OK: puppet ran at Mon Mar 25 11:57:27 UTC 2013 [11:57:37] RECOVERY - Puppet freshness on mw22 is OK: puppet ran at Mon Mar 25 11:57:27 UTC 2013 [11:57:37] RECOVERY - Puppet freshness on pc1003 is OK: puppet ran at Mon Mar 25 11:57:27 UTC 2013 [11:57:38] RECOVERY - Puppet freshness on search20 is OK: puppet ran at Mon Mar 25 11:57:27 UTC 2013 [11:57:38] RECOVERY - Puppet freshness on mw1002 is OK: puppet ran at Mon Mar 25 11:57:27 UTC 2013 [11:57:38] RECOVERY - Puppet freshness on mw31 is OK: puppet ran at Mon Mar 25 11:57:32 UTC 2013 [11:57:47] RECOVERY - Puppet freshness on mw1013 is OK: puppet ran at Mon Mar 25 11:57:40 UTC 2013 [11:57:47] RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Mon Mar 25 11:57:40 UTC 2013 [11:57:47] RECOVERY - Puppet freshness on mw85 is OK: puppet ran at Mon Mar 25 11:57:40 UTC 2013 [11:57:47] RECOVERY - Puppet freshness on mw46 is OK: puppet ran at Mon Mar 25 11:57:40 UTC 2013 [11:57:47] RECOVERY - Puppet freshness on virt7 is OK: puppet ran at Mon Mar 25 11:57:40 UTC 2013 [11:57:58] RECOVERY - Puppet freshness on mw100 is OK: puppet ran at Mon Mar 25 11:57:51 UTC 2013 [11:58:09] RECOVERY - Puppet freshness on mw1036 is OK: puppet ran at Mon Mar 25 11:58:03 UTC 2013 [11:58:17] RECOVERY - Puppet freshness on analytics1001 is OK: puppet ran at Mon Mar 25 11:58:09 UTC 2013 [11:58:17] RECOVERY - Puppet freshness on mw1135 is OK: puppet ran at Mon Mar 25 11:58:14 UTC 2013 [11:58:27] RECOVERY - Puppet freshness on mw87 is OK: puppet ran at Mon Mar 25 11:58:20 UTC 2013 [11:58:27] RECOVERY - Puppet freshness on mw1020 is OK: puppet ran at Mon Mar 25 11:58:20 UTC 2013 [11:58:27] RECOVERY - Puppet freshness on mw1039 is OK: puppet ran at Mon Mar 25 11:58:20 UTC 2013 [11:58:27] RECOVERY - Puppet freshness on search1018 is OK: puppet ran at Mon Mar 25 11:58:25 UTC 2013 [11:58:48] RECOVERY - Puppet freshness on search1015 is OK: puppet ran at Mon Mar 25 11:58:35 UTC 2013 [11:58:49] RECOVERY - Puppet freshness on mw1035 is OK: puppet ran at Mon Mar 25 11:58:42 UTC 2013 [11:58:58] RECOVERY - Puppet freshness on srv273 is OK: puppet ran at Mon Mar 25 11:58:52 UTC 2013 [11:59:08] RECOVERY - Puppet freshness on sq67 is OK: puppet ran at Mon Mar 25 11:58:57 UTC 2013 [11:59:09] RECOVERY - Puppet freshness on mw1027 is OK: puppet ran at Mon Mar 25 11:59:02 UTC 2013 [11:59:17] RECOVERY - Puppet freshness on es1007 is OK: puppet ran at Mon Mar 25 11:59:07 UTC 2013 [11:59:17] RECOVERY - Puppet freshness on srv252 is OK: puppet ran at Mon Mar 25 11:59:08 UTC 2013 [11:59:17] RECOVERY - Puppet freshness on chromium is OK: puppet ran at Mon Mar 25 11:59:15 UTC 2013 [11:59:17] RECOVERY - Puppet freshness on mw1137 is OK: puppet ran at Mon Mar 25 11:59:15 UTC 2013 [11:59:17] RECOVERY - Puppet freshness on srv267 is OK: puppet ran at Mon Mar 25 11:59:15 UTC 2013 [11:59:17] RECOVERY - Puppet freshness on srv246 is OK: puppet ran at Mon Mar 25 11:59:15 UTC 2013 [11:59:18] RECOVERY - Puppet freshness on srv283 is OK: puppet ran at Mon Mar 25 11:59:15 UTC 2013 [11:59:27] RECOVERY - Puppet freshness on virt6 is OK: puppet ran at Mon Mar 25 11:59:20 UTC 2013 [11:59:27] RECOVERY - Puppet freshness on mw94 is OK: puppet ran at Mon Mar 25 11:59:20 UTC 2013 [11:59:27] RECOVERY - Puppet freshness on mw124 is OK: puppet ran at Mon Mar 25 11:59:20 UTC 2013 [11:59:27] RECOVERY - Puppet freshness on srv261 is OK: puppet ran at Mon Mar 25 11:59:20 UTC 2013 [11:59:27] RECOVERY - Puppet freshness on srv288 is OK: puppet ran at Mon Mar 25 11:59:20 UTC 2013 [11:59:27] RECOVERY - Puppet freshness on mw81 is OK: puppet ran at Mon Mar 25 11:59:20 UTC 2013 [11:59:27] RECOVERY - Puppet freshness on srv275 is OK: puppet ran at Mon Mar 25 11:59:20 UTC 2013 [11:59:28] RECOVERY - Puppet freshness on srv264 is OK: puppet ran at Mon Mar 25 11:59:20 UTC 2013 [11:59:28] RECOVERY - Puppet freshness on mw1121 is OK: puppet ran at Mon Mar 25 11:59:20 UTC 2013 [11:59:29] RECOVERY - Puppet freshness on srv270 is OK: puppet ran at Mon Mar 25 11:59:20 UTC 2013 [11:59:29] RECOVERY - Puppet freshness on mw1131 is OK: puppet ran at Mon Mar 25 11:59:20 UTC 2013 [11:59:30] RECOVERY - Puppet freshness on srv263 is OK: puppet ran at Mon Mar 25 11:59:26 UTC 2013 [11:59:30] RECOVERY - Puppet freshness on srv237 is OK: puppet ran at Mon Mar 25 11:59:26 UTC 2013 [11:59:31] RECOVERY - Puppet freshness on analytics1026 is OK: puppet ran at Mon Mar 25 11:59:26 UTC 2013 [11:59:31] RECOVERY - Puppet freshness on srv262 is OK: puppet ran at Mon Mar 25 11:59:26 UTC 2013 [11:59:32] RECOVERY - Puppet freshness on mw102 is OK: puppet ran at Mon Mar 25 11:59:26 UTC 2013 [11:59:37] RECOVERY - Puppet freshness on srv277 is OK: puppet ran at Mon Mar 25 11:59:31 UTC 2013 [11:59:37] RECOVERY - Puppet freshness on srv249 is OK: puppet ran at Mon Mar 25 11:59:31 UTC 2013 [11:59:37] RECOVERY - Puppet freshness on srv265 is OK: puppet ran at Mon Mar 25 11:59:31 UTC 2013 [11:59:37] RECOVERY - Puppet freshness on srv279 is OK: puppet ran at Mon Mar 25 11:59:31 UTC 2013 [11:59:37] RECOVERY - Puppet freshness on srv251 is OK: puppet ran at Mon Mar 25 11:59:31 UTC 2013 [11:59:38] RECOVERY - Puppet freshness on mw107 is OK: puppet ran at Mon Mar 25 11:59:31 UTC 2013 [11:59:38] RECOVERY - Puppet freshness on srv255 is OK: puppet ran at Mon Mar 25 11:59:36 UTC 2013 [11:59:39] RECOVERY - Puppet freshness on srv242 is OK: puppet ran at Mon Mar 25 11:59:36 UTC 2013 [11:59:39] RECOVERY - Puppet freshness on mw83 is OK: puppet ran at Mon Mar 25 11:59:36 UTC 2013 [11:59:40] RECOVERY - Puppet freshness on srv235 is OK: puppet ran at Mon Mar 25 11:59:36 UTC 2013 [11:59:40] RECOVERY - Puppet freshness on mw1190 is OK: puppet ran at Mon Mar 25 11:59:36 UTC 2013 [11:59:41] RECOVERY - Puppet freshness on mw88 is OK: puppet ran at Mon Mar 25 11:59:36 UTC 2013 [11:59:47] RECOVERY - Puppet freshness on mw1174 is OK: puppet ran at Mon Mar 25 11:59:36 UTC 2013 [11:59:47] RECOVERY - Puppet freshness on sq68 is OK: puppet ran at Mon Mar 25 11:59:36 UTC 2013 [11:59:47] RECOVERY - Puppet freshness on srv300 is OK: puppet ran at Mon Mar 25 11:59:41 UTC 2013 [11:59:47] RECOVERY - Puppet freshness on srv256 is OK: puppet ran at Mon Mar 25 11:59:41 UTC 2013 [11:59:47] RECOVERY - Puppet freshness on srv250 is OK: puppet ran at Mon Mar 25 11:59:41 UTC 2013 [11:59:47] RECOVERY - Puppet freshness on mw1198 is OK: puppet ran at Mon Mar 25 11:59:41 UTC 2013 [11:59:48] RECOVERY - Puppet freshness on mw1142 is OK: puppet ran at Mon Mar 25 11:59:41 UTC 2013 [11:59:48] RECOVERY - Puppet freshness on mw105 is OK: puppet ran at Mon Mar 25 11:59:41 UTC 2013 [11:59:49] RECOVERY - Puppet freshness on srv289 is OK: puppet ran at Mon Mar 25 11:59:41 UTC 2013 [11:59:49] RECOVERY - Puppet freshness on srv290 is OK: puppet ran at Mon Mar 25 11:59:41 UTC 2013 [11:59:50] RECOVERY - Puppet freshness on mw92 is OK: puppet ran at Mon Mar 25 11:59:42 UTC 2013 [11:59:50] RECOVERY - Puppet freshness on srv243 is OK: puppet ran at Mon Mar 25 11:59:42 UTC 2013 [11:59:51] RECOVERY - Puppet freshness on mw96 is OK: puppet ran at Mon Mar 25 11:59:42 UTC 2013 [11:59:51] RECOVERY - Puppet freshness on mw116 is OK: puppet ran at Mon Mar 25 11:59:42 UTC 2013 [11:59:52] RECOVERY - Puppet freshness on analytics1027 is OK: puppet ran at Mon Mar 25 11:59:42 UTC 2013 [11:59:52] RECOVERY - Puppet freshness on srv269 is OK: puppet ran at Mon Mar 25 11:59:42 UTC 2013 [11:59:53] RECOVERY - Puppet freshness on mw117 is OK: puppet ran at Mon Mar 25 11:59:42 UTC 2013 [11:59:57] RECOVERY - Puppet freshness on srv291 is OK: puppet ran at Mon Mar 25 11:59:47 UTC 2013 [11:59:57] RECOVERY - Puppet freshness on mw95 is OK: puppet ran at Mon Mar 25 11:59:47 UTC 2013 [11:59:57] RECOVERY - Puppet freshness on srv271 is OK: puppet ran at Mon Mar 25 11:59:47 UTC 2013 [11:59:57] RECOVERY - Puppet freshness on srv248 is OK: puppet ran at Mon Mar 25 11:59:47 UTC 2013 [11:59:57] RECOVERY - Puppet freshness on mw114 is OK: puppet ran at Mon Mar 25 11:59:47 UTC 2013 [12:01:06] RECOVERY - Puppet freshness on srv297 is OK: puppet ran at Mon Mar 25 12:00:47 UTC 2013 [12:01:06] RECOVERY - Puppet freshness on mw1081 is OK: puppet ran at Mon Mar 25 12:00:47 UTC 2013 [12:01:07] RECOVERY - Puppet freshness on mw1075 is OK: puppet ran at Mon Mar 25 12:00:47 UTC 2013 [12:01:07] RECOVERY - Puppet freshness on ms-be10 is OK: puppet ran at Mon Mar 25 12:00:48 UTC 2013 [12:01:08] RECOVERY - Puppet freshness on mw1015 is OK: puppet ran at Mon Mar 25 12:00:48 UTC 2013 [12:01:08] RECOVERY - Puppet freshness on mw1128 is OK: puppet ran at Mon Mar 25 12:00:48 UTC 2013 [12:01:09] RECOVERY - Puppet freshness on analytics1020 is OK: puppet ran at Mon Mar 25 12:00:53 UTC 2013 [12:01:09] RECOVERY - Puppet freshness on mc1004 is OK: puppet ran at Mon Mar 25 12:00:53 UTC 2013 [12:01:10] RECOVERY - Puppet freshness on srv294 is OK: puppet ran at Mon Mar 25 12:00:58 UTC 2013 [12:01:10] RECOVERY - Puppet freshness on mw1031 is OK: puppet ran at Mon Mar 25 12:01:04 UTC 2013 [12:01:28] RECOVERY - Puppet freshness on search30 is OK: puppet ran at Mon Mar 25 12:01:24 UTC 2013 [12:01:47] RECOVERY - Puppet freshness on fenari is OK: puppet ran at Mon Mar 25 12:01:39 UTC 2013 [12:01:57] RECOVERY - Puppet freshness on mw49 is OK: puppet ran at Mon Mar 25 12:01:48 UTC 2013 [12:02:09] RECOVERY - Puppet freshness on mw39 is OK: puppet ran at Mon Mar 25 12:02:05 UTC 2013 [12:02:09] RECOVERY - Puppet freshness on mw33 is OK: puppet ran at Mon Mar 25 12:02:05 UTC 2013 [12:02:18] RECOVERY - Puppet freshness on mw1112 is OK: puppet ran at Mon Mar 25 12:02:10 UTC 2013 [12:03:47] RECOVERY - Puppet freshness on mw1088 is OK: puppet ran at Mon Mar 25 12:03:45 UTC 2013 [12:05:27] RECOVERY - Puppet freshness on mw1052 is OK: puppet ran at Mon Mar 25 12:05:25 UTC 2013 [12:05:59] RECOVERY - Puppet freshness on mw1175 is OK: puppet ran at Mon Mar 25 12:05:48 UTC 2013 [12:05:59] RECOVERY - Puppet freshness on mw1182 is OK: puppet ran at Mon Mar 25 12:05:49 UTC 2013 [12:06:08] RECOVERY - Puppet freshness on mw1114 is OK: puppet ran at Mon Mar 25 12:05:59 UTC 2013 [12:07:07] RECOVERY - Puppet freshness on mw41 is OK: puppet ran at Mon Mar 25 12:06:59 UTC 2013 [12:09:27] RECOVERY - Puppet freshness on mw1115 is OK: puppet ran at Mon Mar 25 12:09:19 UTC 2013 [12:09:27] RECOVERY - Puppet freshness on mw1004 is OK: puppet ran at Mon Mar 25 12:09:26 UTC 2013 [12:10:27] RECOVERY - Puppet freshness on srv244 is OK: puppet ran at Mon Mar 25 12:10:20 UTC 2013 [12:11:08] RECOVERY - Apache HTTP on mw27 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.155 second response time [12:11:57] RECOVERY - Puppet freshness on mw35 is OK: puppet ran at Mon Mar 25 12:11:48 UTC 2013 [12:13:59] RECOVERY - Puppet freshness on mw60 is OK: puppet ran at Mon Mar 25 12:13:54 UTC 2013 [12:16:07] RECOVERY - Puppet freshness on cp3021 is OK: puppet ran at Mon Mar 25 12:16:04 UTC 2013 [12:16:17] PROBLEM - Puppet freshness on mw44 is CRITICAL: Puppet has not run in the last 10 hours [12:16:39] RECOVERY - Puppet freshness on mw1047 is OK: puppet ran at Mon Mar 25 12:16:27 UTC 2013 [12:17:27] New patchset: Mark Bergsma; "Don't backup GeoIP databases in the filebucket" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55565 [12:18:02] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55565 [12:18:17] PROBLEM - Puppet freshness on mw1092 is CRITICAL: Puppet has not run in the last 10 hours [12:18:47] RECOVERY - Puppet freshness on mw36 is OK: puppet ran at Mon Mar 25 12:18:39 UTC 2013 [12:19:17] PROBLEM - Puppet freshness on colby is CRITICAL: Puppet has not run in the last 10 hours [12:20:09] RECOVERY - Puppet freshness on mw53 is OK: puppet ran at Mon Mar 25 12:19:59 UTC 2013 [12:20:10] RECOVERY - Puppet freshness on mw1185 is OK: puppet ran at Mon Mar 25 12:20:04 UTC 2013 [12:20:11] RECOVERY - Puppet freshness on mw1022 is OK: puppet ran at Mon Mar 25 12:20:05 UTC 2013 [12:23:31] New patchset: Matthias Mullie; "Overwrite AFTv5 namespaces on wmflabs" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55566 [12:24:17] PROBLEM - Puppet freshness on mw62 is CRITICAL: Puppet has not run in the last 10 hours [12:25:08] RECOVERY - Puppet freshness on cp3022 is OK: puppet ran at Mon Mar 25 12:25:00 UTC 2013 [12:25:08] RECOVERY - Puppet freshness on mw1119 is OK: puppet ran at Mon Mar 25 12:25:00 UTC 2013 [12:27:19] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55566 [12:28:17] RECOVERY - Puppet freshness on mw32 is OK: puppet ran at Mon Mar 25 12:28:10 UTC 2013 [12:31:57] RECOVERY - Puppet freshness on mw1006 is OK: puppet ran at Mon Mar 25 12:31:48 UTC 2013 [12:32:28] RECOVERY - Puppet freshness on mw1071 is OK: puppet ran at Mon Mar 25 12:32:22 UTC 2013 [12:33:09] RECOVERY - Puppet freshness on mw1207 is OK: puppet ran at Mon Mar 25 12:32:57 UTC 2013 [12:35:04] !log Inserted varnish 3.0.3plus-rc1-wm8 Varnish packages into the APT repository [12:40:34] New patchset: Mark Bergsma; "Upgrade bits Varnish packages to 3.0.3plus-rc1-wm8" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55568 [12:40:49] morebots died [12:41:16] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55568 [12:53:00] New patchset: Mark Bergsma; "Test: allow mobile clients to cache favicon.ico and Gadget js for an hour" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55571 [12:54:30] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55571 [13:10:15] New patchset: Mark Bergsma; "Fix regex" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55573 [13:10:40] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55573 [13:30:17] RECOVERY - Puppet freshness on cp3003 is OK: puppet ran at Mon Mar 25 13:30:08 UTC 2013 [13:35:19] New patchset: Mark Bergsma; "Allow apple-touch icons to be cached by clients" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55577 [13:36:21] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55577 [13:41:27] RECOVERY - Puppet freshness on cp3009 is OK: puppet ran at Mon Mar 25 13:41:21 UTC 2013 [13:44:31] New review: Zfilipin; "It is hard for me to review the code because I do not know the context and I am not familiar with Pu..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/54692 [13:46:21] RECOVERY - Puppet freshness on stat1001 is OK: puppet ran at Mon Mar 25 13:46:14 UTC 2013 [13:52:20] PROBLEM - Puppet freshness on db67 is CRITICAL: Puppet has not run in the last 10 hours [13:53:17] PROBLEM - Puppet freshness on ms-fe2 is CRITICAL: Puppet has not run in the last 10 hours [13:56:42] New patchset: Diederik; "Added domain referer info to blog query, and ignore search and preview urls." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55390 [13:59:23] New review: Milimetric; "if someone with +2 could please merge, it looks good" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/55390 [13:59:51] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55390 [14:02:09] PROBLEM - SSH on amslvs1 is CRITICAL: Server answer: [14:04:07] RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [14:12:07] New patchset: Ottomata; "Disabling gerrit stats for now, it is broken :/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55581 [14:12:44] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55581 [14:14:17] PROBLEM - Puppet freshness on mw1126 is CRITICAL: Puppet has not run in the last 10 hours [14:15:17] PROBLEM - Puppet freshness on mw1099 is CRITICAL: Puppet has not run in the last 10 hours [14:17:17] PROBLEM - Puppet freshness on mw1152 is CRITICAL: Puppet has not run in the last 10 hours [14:17:18] PROBLEM - Puppet freshness on mw69 is CRITICAL: Puppet has not run in the last 10 hours [14:17:44] New review: Silke Meyer; "The wikidata_singlenode module doesn't seem to know about mediawiki_singlenode/manifests/mw-extensio..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/51797 [14:18:19] PROBLEM - Puppet freshness on mw1001 is CRITICAL: Puppet has not run in the last 10 hours [14:18:19] PROBLEM - Puppet freshness on mw1103 is CRITICAL: Puppet has not run in the last 10 hours [14:19:19] PROBLEM - Puppet freshness on mw1025 is CRITICAL: Puppet has not run in the last 10 hours [14:19:19] PROBLEM - Puppet freshness on mw1109 is CRITICAL: Puppet has not run in the last 10 hours [14:20:17] PROBLEM - Puppet freshness on grosley is CRITICAL: Puppet has not run in the last 10 hours [14:20:17] PROBLEM - Puppet freshness on locke is CRITICAL: Puppet has not run in the last 10 hours [14:20:17] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [14:20:17] PROBLEM - Puppet freshness on mw1116 is CRITICAL: Puppet has not run in the last 10 hours [14:20:17] PROBLEM - Puppet freshness on mw1155 is CRITICAL: Puppet has not run in the last 10 hours [14:20:18] PROBLEM - Puppet freshness on tola is CRITICAL: Puppet has not run in the last 10 hours [14:22:21] PROBLEM - Puppet freshness on hooft is CRITICAL: Puppet has not run in the last 10 hours [14:22:21] PROBLEM - Puppet freshness on ms6 is CRITICAL: Puppet has not run in the last 10 hours [14:22:21] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [14:23:17] PROBLEM - Puppet freshness on mw1012 is CRITICAL: Puppet has not run in the last 10 hours [14:23:17] PROBLEM - Puppet freshness on mw48 is CRITICAL: Puppet has not run in the last 10 hours [14:24:19] PROBLEM - Puppet freshness on mw1063 is CRITICAL: Puppet has not run in the last 10 hours [14:50:24] New patchset: Mark Bergsma; "Set default_ttl for esams upload to just 1 day" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55588 [14:51:13] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55588 [14:52:35] New patchset: Ottomata; "Sending nginx logs to their own udp2log instance on gadolinium." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55394 [14:53:34] New patchset: Ottomata; "Sending nginx logs to their own udp2log instance on gadolinium." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55394 [14:55:47] New patchset: Ottomata; "Sending nginx logs to their own udp2log instance on gadolinium." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55394 [14:57:51] New patchset: Ottomata; "Sending nginx logs to their own udp2log instance on gadolinium." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55394 [14:58:54] xyzram: speaking publicly there will grant more people attention :-] [14:59:37] ^demon|nopower: can we (you, xyzram, I) sudo as `lsearch` user on the search boxes ? Or is that restricted to ops ? [14:59:47] cause we could use the rights to restart the incremental updater :-] [14:59:58] hashar: sure, I saw your puppet patch but I guess it hasn't yet been merged yet, right? [15:00:02] (which I thought was running in a while(true) loop) [15:00:19] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55394 [15:00:35] hashar! thanks for your jenkins puppet typo checker! [15:00:41] it just caught a 'gadolinum' typo :) [15:00:48] hashar: Yes it is but even the shell script seems to die. [15:01:56] <^demon|nopower> xyzram, hashar: Link to patch? I can't find it. [15:02:18] https://gerrit.wikimedia.org/r/#/c/55406/1/manifests/search.pp [15:02:49] ahh [15:03:03] so yeah that patch is an attempt to create an upstart job for the inc-updater [15:03:13] talked about that with peter last friday [15:03:16] <^demon|nopower> Ah, I saw that. I thought you meant a patch for letting people sudo to lsearch. [15:03:19] or earlier can't remember [15:03:34] what I would love [15:03:45] is to have a lsearchadmin administrative account [15:03:55] New patchset: Ottomata; "Fixing udp2log define usage" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55592 [15:04:09] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55592 [15:05:29] ^demon|nopower: No, sorry, that was a continuation of a private chat with hashar. [15:05:42] don't we have a nagios check to ensure the process is running? [15:05:54] ah [15:05:56] might not work [15:05:56] I'm not able to restart the incremental updater. [15:06:49] Wonder why it keeps dying ? [15:07:47] no idea [15:07:51] there is definitely a while true [15:08:46] Tim noted that it died while an import was happening, so it may be that the import is locking files causing failure. [15:09:17] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [15:09:21] I guess you will want to debug it with Peter [15:09:38] and talk with him about how you could be granted some specific rights to restart the inc-updater [15:09:48] So if there is some import job scheduled, that needs to maybe stop the inc. updater, do the import and restart it. [15:12:04] !log Lucene search incremental updater died on searchidx1001 [15:12:21] so we need some root to run on searchidx1001 : su -s /bin/bash -c "/a/search/lucene.jobs.sh inc-updater-start" lsearch [15:13:09] New patchset: Ottomata; "Saving nginx logs in separate directory to avoid udp2log::instance conflicts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55593 [15:14:06] New patchset: Ottomata; "Saving nginx logs in separate directory to avoid udp2log::instance conflicts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55593 [15:14:31] xyzram, could you reply to my email about merging Search components in Bugzilla at some point? Nothing urgent, but would love to get this off my list. Thanks in advance :) [15:14:37] New patchset: Matthias Mullie; "On labs, there's no dedicated cluster for AFTv5's data; false will default to main MW db" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55594 [15:14:46] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55593 [15:14:46] <^demon|nopower> xyzram: Soo, I was thinking of two things. 1) Do we have a central project page for Solr yet? If not, I was going to start collecting requirements. [15:15:03] <^demon|nopower> And 2) Did you have any plans for getting better error reporting from lsearchd back to Special:Search? [15:15:26] <^demon|nopower> (I figure those were the two big things to tackle this week, minus any other firefighting) [15:15:28] andre__: Thought I did about 32min ago. [15:16:24] ^demon|nopower: No Solr page, but great idea to start one. [15:16:59] RECOVERY - Puppet freshness on gadolinium is OK: puppet ran at Mon Mar 25 15:16:55 UTC 2013 [15:17:57] RECOVERY - udp2log log age for gadolinium on gadolinium is OK: OK: all log files active [15:19:11] I was working on (2) but got bogged down in the code logic. Some of the functions in MC (RMIMessengerClient) swallow networking failures but they do output warnings ... [15:19:58] Over the weekend I was able to again reproduce the "zero results" issue with the script in the ticket (with the wikidata URL) ... [15:20:18] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55594 [15:21:03] But couldn't find the warning messages in the logs after a quick search but I'm going to look again this morning. [15:22:29] <^demon|nopower> https://www.mediawiki.org/wiki/Solr - base page started. [15:22:35] The global configuration which drives most of the behavior is rather complex (some 8000 IndexIds !) so I'm going to push a patch that dumps the entire config data structures to the log files. [15:22:58] New patchset: Ottomata; "Fixing ownership of fundraising log dirs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55596 [15:23:18] <^demon|nopower> Yeah, that global configuration is a mess. [15:23:33] xyzram: Eeks, I'm very sorry. Mid-air collision. :-/ [15:24:25] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55596 [15:28:46] New patchset: Ottomata; "Saving 1/100 of nginx logs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55600 [15:31:58] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55600 [15:47:26] who said Solr?:) [15:47:51] <^demon> Yessir. [15:48:59] ^demon, I heard all the Solr stuff was going to be passed to you, including the already deployed bits [15:49:11] <^demon> Hahahaha [15:49:49] <^demon> You must've been lied to :p [15:50:39] yeh:) [15:51:06] anyway, do you need any help with this? [15:53:20] New patchset: Ottomata; "Adding puppet Limn module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49710 [15:54:32] <^demon> Yes, actually. Step 1 is making sure we understand what everybody's needs are. [15:54:53] <^demon> So we can design a system that suits everyone, rather than each project installing their own solr for their own purpose :) [15:55:14] dreamers!:P [15:55:33] <^demon> Yeah, well, shoot for the stars...maybe I'll get the moon. [15:55:43] New review: Ottomata; "(8 comments)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49710 [15:55:45] <^demon> https://www.mediawiki.org/wiki/Solr#For_other_WMF_applications - anything in here you could clarify would be useful. [15:55:59] <^demon> s/clarify/add/ -- this was just started. [15:56:35] have you seen https://wikitech.wikimedia.org/wiki/Solr ? [15:56:50] <^demon> I had not yet. [15:57:49] <^demon> Hmm, so everybody using solr is already using those same puppet classes? [15:57:57] yes [15:57:58] <^demon> happy days :) [15:58:48] anything opsy you want to se on wikitech? [16:00:14] <^demon> Lemme finish my lunch and I'll take a closer look. [16:02:00] morebots died. [16:03:09] PROBLEM - HTTP on fenari is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:04:06] RECOVERY - HTTP on fenari is OK: HTTP OK: HTTP/1.1 200 OK - 4915 bytes in 4.770 second response time [16:04:36] RECOVERY - Puppet freshness on cp1023 is OK: puppet ran at Mon Mar 25 16:04:28 UTC 2013 [16:09:16] RECOVERY - Puppet freshness on cp1021 is OK: puppet ran at Mon Mar 25 16:09:09 UTC 2013 [16:10:26] RECOVERY - Puppet freshness on cp1022 is OK: puppet ran at Mon Mar 25 16:10:18 UTC 2013 [16:12:36] RECOVERY - Puppet freshness on cp1024 is OK: puppet ran at Mon Mar 25 16:12:28 UTC 2013 [16:13:36] RECOVERY - Puppet freshness on cp1025 is OK: puppet ran at Mon Mar 25 16:13:27 UTC 2013 [16:14:16] PROBLEM - Puppet freshness on mw1077 is CRITICAL: Puppet has not run in the last 10 hours [16:14:36] RECOVERY - Puppet freshness on cp1026 is OK: puppet ran at Mon Mar 25 16:14:29 UTC 2013 [16:15:36] RECOVERY - Puppet freshness on cp1027 is OK: puppet ran at Mon Mar 25 16:15:27 UTC 2013 [16:16:16] PROBLEM - Puppet freshness on mw1104 is CRITICAL: Puppet has not run in the last 10 hours [16:16:36] RECOVERY - Puppet freshness on cp1028 is OK: puppet ran at Mon Mar 25 16:16:34 UTC 2013 [16:17:13] New patchset: Hashar; "create jenkins user with systemuser" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53880 [16:17:13] New patchset: Hashar; "puppet now manage jenkins ssh authorized_keys" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53736 [16:17:13] New patchset: Hashar; "systemuser learned 'managehome' (default true)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53879 [16:17:36] RECOVERY - Puppet freshness on cp1029 is OK: puppet ran at Mon Mar 25 16:17:29 UTC 2013 [16:18:36] RECOVERY - Puppet freshness on cp1030 is OK: puppet ran at Mon Mar 25 16:18:28 UTC 2013 [16:19:16] PROBLEM - Puppet freshness on mw1043 is CRITICAL: Puppet has not run in the last 10 hours [16:19:37] ^demon, https://www.mediawiki.org/wiki/Solr#GeoData [16:20:16] PROBLEM - Puppet freshness on mw1080 is CRITICAL: Puppet has not run in the last 10 hours [16:20:26] RECOVERY - Puppet freshness on cp1032 is OK: puppet ran at Mon Mar 25 16:20:21 UTC 2013 [16:21:26] RECOVERY - Puppet freshness on cp1033 is OK: puppet ran at Mon Mar 25 16:21:24 UTC 2013 [16:22:16] PROBLEM - Puppet freshness on mw1003 is CRITICAL: Puppet has not run in the last 10 hours [16:22:36] RECOVERY - Puppet freshness on cp1034 is OK: puppet ran at Mon Mar 25 16:22:28 UTC 2013 [16:24:30] RECOVERY - Puppet freshness on cp1036 is OK: puppet ran at Mon Mar 25 16:24:20 UTC 2013 [16:25:11] New review: coren; "I got a 1 on my Knowledge (Git) roll and seem to have submitted a partial patch. Or something. New..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53587 [16:25:16] PROBLEM - Puppet freshness on virt5 is CRITICAL: Puppet has not run in the last 10 hours [16:26:16] PROBLEM - Puppet freshness on mw1089 is CRITICAL: Puppet has not run in the last 10 hours [16:27:18] PROBLEM - Puppet freshness on mw1129 is CRITICAL: Puppet has not run in the last 10 hours [16:29:12] New patchset: Ottomata; "Adding puppet Limn module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49710 [16:30:16] PROBLEM - Puppet freshness on mw1141 is CRITICAL: Puppet has not run in the last 10 hours [16:30:33] New patchset: Aude; "Update Wikibase settings" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55605 [16:30:47] New patchset: coren; "New toollabs:: class to config Tool Labs servers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53587 [16:31:03] <^demon> MaxSem: Thanks. [16:31:18] ^demon, is that enough? [16:31:25] <^demon> For now, plus I have the other page. [16:31:40] <^demon> This may sound like a stupid question...but multiple indices can run on the same node, right? [16:31:56] yes [16:32:06] if by indices you mean cores [16:32:27] <^demon> Probably yes. [16:32:50] * ^demon reads https://wiki.apache.org/solr/SolrTerminology [16:34:39] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55500 [16:36:03] New review: coren; "Ignore this changeset; git reviewed too early." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53587 [16:37:49] New patchset: Ottomata; "Adding puppet Limn module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49710 [16:37:51] New review: Dzahn; "recheck" [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/55462 [16:38:17] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: Puppet has not run in the last 10 hours [16:38:17] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [16:38:17] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: Puppet has not run in the last 10 hours [16:38:17] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: Puppet has not run in the last 10 hours [16:38:17] PROBLEM - Puppet freshness on analytics1003 is CRITICAL: Puppet has not run in the last 10 hours [16:41:04] New patchset: Ottomata; "Adding puppet Limn module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49710 [16:41:40] <^demon> MaxSem: Did you look at the pecl solr extension as well? [16:41:48] yes [16:41:56] it's too primitive [16:42:06] <^demon> Looked like it, was just wondering. [16:42:30] Solarium is a nice beast [16:42:37] although a bit crufty [16:42:44] New patchset: Ottomata; "Adding puppet Limn module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49710 [16:43:11] New patchset: Dzahn; "Merge "Added Edward Baker to English Planet" into production" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55606 [16:45:46] Change abandoned: Dzahn; "was supposed to be amended to 55462" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55606 [16:49:42] ops team / Tim-away : Could somebody take a look at https://bugzilla.wikimedia.org/show_bug.cgi?id=46530 please? (Search index not updating on en.wikipedia.org) [16:52:26] From: http://wiki.apache.org/solr/SolrTerminology it looks like a Solr Core is an instance of a class that acts as a low-level interface to a single index. [16:52:46] New patchset: Dzahn; "Added Yuvi Panda to English Planet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55462 [16:53:48] !log torrus deadlocked, restarting apache & recompiling its xml [16:54:17] xyzram, core = index + config [16:54:26] (in a practical sense) [16:55:49] andre__: There was some discussion of that issue earlier here and also on Friday last week. We (Chad, Andoine, me) need privileges to restart the IncrementalUpdater. [16:56:04] New review: Ottomata; "I fixed most of your previous comments, thanks!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49710 [16:56:14] Also need to figure out why it keeps dying. [16:56:30] xyzram, oh, I see. That's already some info, thanks. [16:56:39] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55462 [16:56:50] New review: Ottomata; "Well that previous formatting didn't work well :(. Let me try again:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/49710 [16:57:43] phew, alllright! paravoid, I'm ready for a new limn module review, whenver you got da time [16:57:45] https://gerrit.wikimedia.org/r/#/c/49710/ [16:58:42] MaxSem: whats in the config ? [17:00:05] virtually every aspect of Solr is configurable by core. also schema, stemming etc [17:09:03] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55605 [17:15:12] New patchset: Ottomata; "Disabling packet loss monitor for nginx udp2log instance" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55609 [17:18:19] !log authdns-update [17:20:45] <^demon> MaxSem: Have you played with the zookeeper stuff in 4.x yet? [17:21:06] no [17:21:14] <^demon> Looks interesting. [17:24:45] !log reedy synchronized wmf-config/ [17:24:51] Logged the message, Master [17:26:38] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55609 [17:28:41] ottomata: gadolinium be spamming, yo [17:28:44] File /a/log/nginx/packet-loss.log cannot be read. [17:28:46] working on it! [17:28:51] https://gerrit.wikimedia.org/r/#/c/55609/ [17:28:51] :( [17:28:53] :) [17:29:20] ottomata: you missed a "kthxbye" with those commits [17:29:34] it's broken, we don't have time nor interest, who cares [17:29:40] kthxbye :) [17:30:08] haha [17:30:56] <^demon> paravoid: So, I think the new solr 4.x + zookeeper does a lot of the sharding stuff we were liking in elasticsearch. [17:31:04] * paravoid hands a carpet to ottomata  [17:31:08] quick, hide the nginx module under it [17:31:12] <^demon> I'm going to setup a mini-cluster in labs and see how it works in practice. [17:31:42] yup you know, gotta prioritize! [17:31:48] no offence intended and sorry for the snarkiness, I just think this approach sucks a lot :) [17:31:51] New patchset: coren; "New toollabs:: class to config Tool Labs servers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/53587 [17:31:55] paravoid: is ceph down? [17:32:00] (yeah, none taken, plenty of humor came along with it) [17:32:14] AaronSchulz: apparently so... [17:32:19] paravoid, I agree, its not the best approach, but we gain very little for the amount of time it would take to make it work correctly [17:32:53] sbernardin: ping me when you have racked and cfg'd rdb1/2 [17:33:14] AaronSchulz: (looking) [17:35:16] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: Puppet has not run in the last 10 hours [17:35:16] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [17:35:16] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: Puppet has not run in the last 10 hours [17:35:16] PROBLEM - Puppet freshness on msfe1002 is CRITICAL: Puppet has not run in the last 10 hours [17:35:37] OK [17:36:43] cmjohnson1: that's probably going to be on Thursday since Fedex usually comes late [17:37:07] ok, i thoght they were already there [17:38:22] Nope...shipment is scheduled to arrive on Wednesday [17:39:49] cmjohnson1: https://rt.wikimedia.org/Ticket/Display.html?id=4712 [17:41:45] who buys the domains? [17:43:22] sup? [17:43:30] A few of us can, but I do most of them [17:44:29] Platonides: ^ =] [17:49:13] New patchset: Demon; "Revert "Updating gerrit to 2.6-rc0-7-g6e5cc39"" [operations/debs/gerrit] (master) - https://gerrit.wikimedia.org/r/55610 [17:49:35] Change merged: Demon; [operations/debs/gerrit] (master) - https://gerrit.wikimedia.org/r/55610 [17:55:56] <^demon> xyzram: Well, here's your NPE: http://p.defau.lt/?bHDeol2_D_i7u7iczvLziQ [17:56:05] <^demon> Yay, let's just always pass null! [17:58:41] New patchset: awjrichards; "Enable mobile login handshake on betalabs" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55611 [18:01:58] Change merged: awjrichards; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55611 [18:06:57] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf12 [18:07:03] Logged the message, Master [18:08:47] Reedy: poke [18:09:31] ? [18:09:39] do you need me to submit a patch to update our submodule? [18:10:17] wtf. Request: GET http://en.wikipedia.org/wiki/Special:Contributions/Aude, from 10.64.0.127 via cp1015.eqiad.wmnet (squid/2.7.STABLE9) to () [18:10:17] I don't need you to, but it'd be appreciated if you could [18:10:20] Error: ERR_CANNOT_FORWARD, errno [No Error] at Mon, 25 Mar 2013 18:09:58 GMT [18:10:26] PROBLEM - Apache HTTP on mw1079 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:10:26] PROBLEM - Apache HTTP on mw1036 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:10:26] PROBLEM - Apache HTTP on mw1066 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:10:26] PROBLEM - Apache HTTP on mw1064 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:10:26] PROBLEM - Apache HTTP on mw1105 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:10:35] Reedy: ok, doing (and wonder why i can't seem user page ^) [18:10:47] back [18:10:53] Was a bit slow first time I loaded it [18:10:56] ok [18:11:16] do config changes for betalabs get deployed automatically? [18:11:39] RECOVERY - Puppet freshness on labstore2 is OK: puppet ran at Mon Mar 25 18:11:05 UTC 2013 [18:11:59] PROBLEM - Apache HTTP on mw1167 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:59] PROBLEM - Apache HTTP on mw1183 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:59] PROBLEM - Apache HTTP on mw1168 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:59] PROBLEM - Apache HTTP on mw1174 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:11:59] PROBLEM - Apache HTTP on mw1054 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:12:24] did anything just get deployed? [18:12:33] RECOVERY - Puppet freshness on labstore3 is OK: puppet ran at Mon Mar 25 18:12:26 UTC 2013 [18:12:36] notpeter: enwiki updated? [18:12:42] Reedy: ^^ [18:12:48] * aude seeing errors, but then hit refresh and i get enwiki [18:12:49] PROBLEM - LVS HTTP IPv4 on m.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - pattern not found - 864 bytes in 0.002 second response time [18:13:06] er? [18:13:06] also on wikidata [18:13:08] you guys on this? i know y'all are in a meeting [18:13:13] Reedy: ???? [18:13:14] you there ? [18:13:14] Request: POST http://www.mediawiki.org/w/index.php?title=Wikimedia_Apps/Commons&action=submit, from 10.64.0.135 via cp1015.eqiad.wmnet (squid/2.7.STABLE9) to () [18:13:15] Error: ERR_CANNOT_FORWARD, errno [No Error] at Mon, 25 Mar 2013 18:11:24 GMT [18:13:16] PROBLEM - LVS HTTP IPv4 on wikinews-lb.pmtpa.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.0 504 Gateway Time-out - 3406 bytes in 0.086 second response time [18:13:18] Request: GET http://www.wikidata.org/wiki/Special:Contributions/Aude, from 10.64.0.136 via cp1015.eqiad.wmnet (squid/2.7.STABLE9) to () [18:13:19] meeting is now about this [18:13:24] Error: ERR_CANNOT_FORWARD, errno [No Error] at Mon, 25 Mar 2013 18:12:13 GMT [18:13:25] I can revert it.. [18:13:26] PROBLEM - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:26] PROBLEM - LVS HTTPS IPv4 on wikinews-lb.pmtpa.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:26] PROBLEM - LVS HTTP IPv4 on wikinews-lb.esams.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:13:29] okay thanks [18:13:31] yes please [18:13:52] Reedy: every anon request is hitting the apaches with GET /wiki/Special:BannerRandom?userlang=en&sitename=Wikipedia&project=wikipedia&anonymous=true&bucket=0&country=ID&device=desktop&slot&_=1364235170845 [18:14:03] anonymous=true heh [18:14:27] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rb [18:14:28] it seems like the query param named _ is given an ever changing unix timestamp as its value [18:14:29] oh noes [18:14:33] :( [18:14:33] Logged the message, Master [18:14:46] Well, that's a FR issue... [18:15:04] it's a banner? [18:15:16] PROBLEM - Puppet freshness on analytics1010 is CRITICAL: Puppet has not run in the last 10 hours [18:15:16] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: Puppet has not run in the last 10 hours [18:15:16] PROBLEM - Puppet freshness on db1052 is CRITICAL: Puppet has not run in the last 10 hours [18:15:16] PROBLEM - Puppet freshness on es10 is CRITICAL: Puppet has not run in the last 10 hours [18:15:16] PROBLEM - Puppet freshness on mc1005 is CRITICAL: Puppet has not run in the last 10 hours [18:15:16] PROBLEM - Puppet freshness on mw1017 is CRITICAL: Puppet has not run in the last 10 hours [18:15:16] PROBLEM - Puppet freshness on mw1018 is CRITICAL: Puppet has not run in the last 10 hours [18:15:17] PROBLEM - Puppet freshness on mw1144 is CRITICAL: Puppet has not run in the last 10 hours [18:15:17] PROBLEM - Puppet freshness on mw1147 is CRITICAL: Puppet has not run in the last 10 hours [18:15:18] PROBLEM - Puppet freshness on mw1214 is CRITICAL: Puppet has not run in the last 10 hours [18:15:18] PROBLEM - Puppet freshness on mw1217 is CRITICAL: Puppet has not run in the last 10 hours [18:15:19] PROBLEM - Puppet freshness on mw14 is CRITICAL: Puppet has not run in the last 10 hours [18:15:19] PROBLEM - Puppet freshness on mw16 is CRITICAL: Puppet has not run in the last 10 hours [18:15:20] PROBLEM - Puppet freshness on mw5 is CRITICAL: Puppet has not run in the last 10 hours [18:15:20] PROBLEM - Puppet freshness on mw9 is CRITICAL: Puppet has not run in the last 10 hours [18:15:21] yes [18:15:21] PROBLEM - Puppet freshness on search1010 is CRITICAL: Puppet has not run in the last 10 hours [18:15:21] PROBLEM - Puppet freshness on search1008 is CRITICAL: Puppet has not run in the last 10 hours [18:15:22] PROBLEM - Puppet freshness on search1014 is CRITICAL: Puppet has not run in the last 10 hours [18:15:22] PROBLEM - Puppet freshness on search1019 is CRITICAL: Puppet has not run in the last 10 hours [18:15:23] PROBLEM - Puppet freshness on search17 is CRITICAL: Puppet has not run in the last 10 hours [18:15:23] PROBLEM - Puppet freshness on search13 is CRITICAL: Puppet has not run in the last 10 hours [18:15:24] ....:...GET /wiki/Special:BannerRandom?userlang=en&sitename=Wikipedia&project=wikipedia&anonymous=true&bucket=0&country=MK&device=desktop&slot=4&_=1364235306493 HTTP/1.0 [18:15:24] icinga-wm: stfu [18:15:24] Host: meta.wikimedia.org [18:15:24] PROBLEM - Puppet freshness on search14 is CRITICAL: Puppet has not run in the last 10 hours [18:15:24] PROBLEM - Puppet freshness on stafford is CRITICAL: Puppet has not run in the last 10 hours [18:15:25] PROBLEM - Puppet freshness on zirconium is CRITICAL: Puppet has not run in the last 10 hours [18:15:26] please kill it [18:15:40] * aude wonders which one [18:15:46] PROBLEM - LVS HTTP IPv6 on wikidata-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - 3447 bytes in 0.072 second response time [18:15:52] PROBLEM - LVS HTTP IPv4 on wikidata-lb.pmtpa.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.0 504 Gateway Time-out - 3380 bytes in 0.069 second response time [18:15:54] does this need a squid ban? [18:16:06] mark: that would fix it [18:16:24] it would be nice not to need to, but I don't know what's hitting it ;) [18:16:25] it wouldn't fix mobile [18:16:33] paravoid: yes it will [18:16:42] RECOVERY - Puppet freshness on search1008 is OK: puppet ran at Mon Mar 25 18:16:27 UTC 2013 [18:16:47] is this just on the desktop site? [18:16:51] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: Puppet has not run in the last 10 hours [18:16:52] PROBLEM - Puppet freshness on kuo is CRITICAL: Puppet has not run in the last 10 hours [18:16:52] PROBLEM - Puppet freshness on ms-be1012 is CRITICAL: Puppet has not run in the last 10 hours [18:16:52] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: Puppet has not run in the last 10 hours [18:16:52] PROBLEM - Puppet freshness on mw1125 is CRITICAL: Puppet has not run in the last 10 hours [18:16:52] PROBLEM - Puppet freshness on db43 is CRITICAL: Puppet has not run in the last 10 hours [18:16:52] PROBLEM - Puppet freshness on mw1048 is CRITICAL: Puppet has not run in the last 10 hours [18:16:53] PROBLEM - Puppet freshness on search19 is CRITICAL: Puppet has not run in the last 10 hours [18:17:10] paravoid: the desktop and mobile site are dependent on the same apache pool [18:17:38] I'm aware of that :) [18:17:55] so that Special is just on meta.wm.org [18:17:59] ok [18:18:09] do i need to walk up to six and poke fundraising people? [18:18:19] someone should [18:18:20] i'm on their irc channel [18:18:21] apparently [18:18:22] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [18:18:55] * aude can turn off the banners [18:19:01] blindly [18:19:03] RECOVERY - Apache HTTP on mw1079 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.051 second response time [18:19:03] RECOVERY - Apache HTTP on mw1185 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.054 second response time [18:19:04] RECOVERY - Apache HTTP on mw1035 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.057 second response time [18:19:04] RECOVERY - Apache HTTP on mw1041 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.065 second response time [18:19:04] RECOVERY - Apache HTTP on mw1066 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.870 second response time [18:19:49] okay, they're on it [18:19:49] i assume they reverted? [18:20:09] seems so [18:20:24] I don't see those requests anymore [18:20:43] down now [18:20:56] i still see some but a lot less [18:20:56] en.m.wikipedia.org seems to be working again [18:21:06] diabled the banners [18:21:09] if that helps [18:21:20] oh, theres more [18:21:22] binasher: I saw some but with no _=epoch [18:22:03] just saw eGET /wiki/Special:BannerRandom?userlang=en&sitename=Wikipedia&project=wikipedia&anonymous=true&bucket=0&country=CA&device=desktop&slot=29&_=1364235689216 [18:22:07] but still, a lot less [18:22:30] binasher: can you tell me more about what's going on? [18:22:31] yeah, there were none on my 5000 packets sample [18:22:35] diabled more [18:23:07] RECOVERY - Puppet freshness on search1010 is OK: puppet ran at Mon Mar 25 18:22:59 UTC 2013 [18:23:17] PROBLEM - Puppet freshness on mw1093 is CRITICAL: Puppet has not run in the last 10 hours [18:23:30] notpeter: IncrementalUpdater died again; can we (Chad, Antoine, me) get privileges to restart it ? [18:24:03] * aude goes back to prepare our submodule [18:24:09] mwalker: anon requests getting through to the apaches (bad) because of this param in the banner requests: _= some random thing that apprently continuously changes [18:24:21] timestamp os something [18:24:25] *or [18:24:40] damnit -- I have no idea where that comes from? [18:24:41] ... [18:24:46] can we just disable CN on the cluster [18:24:52] it's either that or we revert to something good [18:25:19] and I have permissions to do neither [18:25:19] aude turned off all the banners I guess [18:25:24] it's not a banner thing [18:25:26] i think we got them all [18:25:34] aude: thanks. [18:25:35] sure it could be an issue with the extnsion [18:25:44] he requests are for Special:BannerRandom [18:25:48] RECOVERY - Puppet freshness on mw58 is OK: puppet ran at Mon Mar 25 18:25:42 UTC 2013 [18:25:57] yes; Special:BannerRandom is called before the banner is delivered [18:26:04] xyzram: yeah. can you make a ticket and assign it to me? (just so I don't forget...) [18:26:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.392 second response time [18:26:19] ok well if we don't serve banners then it won't be called, which is good enough for this instant [18:26:57] RECOVERY - Puppet freshness on mw55 is OK: puppet ran at Mon Mar 25 18:26:50 UTC 2013 [18:26:57] RECOVERY - Puppet freshness on mw59 is OK: puppet ran at Mon Mar 25 18:26:50 UTC 2013 [18:26:57] RECOVERY - Puppet freshness on mw56 is OK: puppet ran at Mon Mar 25 18:26:55 UTC 2013 [18:26:57] RECOVERY - Puppet freshness on mw57 is OK: puppet ran at Mon Mar 25 18:26:55 UTC 2013 [18:27:07] RECOVERY - Puppet freshness on mw1028 is OK: puppet ran at Mon Mar 25 18:27:01 UTC 2013 [18:27:07] RECOVERY - Puppet freshness on mw1154 is OK: puppet ran at Mon Mar 25 18:27:01 UTC 2013 [18:27:07] RECOVERY - Puppet freshness on mw1058 is OK: puppet ran at Mon Mar 25 18:27:01 UTC 2013 [18:27:07] RECOVERY - Puppet freshness on mw1048 is OK: puppet ran at Mon Mar 25 18:27:01 UTC 2013 [18:27:07] RECOVERY - Puppet freshness on mw1151 is OK: puppet ran at Mon Mar 25 18:27:01 UTC 2013 [18:27:08] RECOVERY - Puppet freshness on mw1026 is OK: puppet ran at Mon Mar 25 18:27:01 UTC 2013 [18:27:08] RECOVERY - Puppet freshness on mw1059 is OK: puppet ran at Mon Mar 25 18:27:01 UTC 2013 [18:27:09] RECOVERY - Puppet freshness on mw1125 is OK: puppet ran at Mon Mar 25 18:27:01 UTC 2013 [18:27:09] RECOVERY - Puppet freshness on mw1138 is OK: puppet ran at Mon Mar 25 18:27:01 UTC 2013 [18:27:10] RECOVERY - Puppet freshness on mw1147 is OK: puppet ran at Mon Mar 25 18:27:01 UTC 2013 [18:27:17] RECOVERY - Puppet freshness on mw1051 is OK: puppet ran at Mon Mar 25 18:27:06 UTC 2013 [18:27:17] RECOVERY - Puppet freshness on mw1050 is OK: puppet ran at Mon Mar 25 18:27:06 UTC 2013 [18:27:17] RECOVERY - Puppet freshness on mw1082 is OK: puppet ran at Mon Mar 25 18:27:06 UTC 2013 [18:27:17] RECOVERY - Puppet freshness on mw1156 is OK: puppet ran at Mon Mar 25 18:27:06 UTC 2013 [18:27:17] RECOVERY - Puppet freshness on mw1144 is OK: puppet ran at Mon Mar 25 18:27:06 UTC 2013 [18:27:18] RECOVERY - Puppet freshness on mw1136 is OK: puppet ran at Mon Mar 25 18:27:06 UTC 2013 [18:27:18] RECOVERY - Puppet freshness on mw1061 is OK: puppet ran at Mon Mar 25 18:27:06 UTC 2013 [18:27:19] RECOVERY - Puppet freshness on mw1053 is OK: puppet ran at Mon Mar 25 18:27:07 UTC 2013 [18:27:19] RECOVERY - Puppet freshness on mw1078 is OK: puppet ran at Mon Mar 25 18:27:07 UTC 2013 [18:28:07] RECOVERY - Puppet freshness on mw12 is OK: puppet ran at Mon Mar 25 18:28:03 UTC 2013 [18:28:25] I can globally disable CN for now if that's what is wanted? [18:28:28] Reedy: whenever you are ready for more deployment https://gerrit.wikimedia.org/r/#/c/55616/ [18:28:38] Reedy: probably good idea [18:28:38] Reedy: yes; disable it [18:28:45] wmgUseCentralNotice = false [18:28:50] yep [18:29:04] line 4060 in InitializeSettings.php [18:29:13] mwalker: aude: still seeing around 300 reqs/sec for meta.wikimedia.org/wiki/Special:BannerRandom with &_=timestamp [18:29:22] binasher: don't know if i got them all [18:29:32] so it isn't really entirely off.. but might just be browser cached js, etc? [18:29:33] mwalker got more of them [18:29:40] could be [18:29:55] we got all of them off [18:30:20] !log reedy synchronized wmf-config/InitialiseSettings.php 'Disable CN everywhere' [18:30:27] Logged the message, Master [18:30:48] New patchset: Reedy; "Globally disable CentralNotice" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55617 [18:31:00] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55617 [18:31:16] binasher, aude; there is a 15 minute cache on banners -- but this is not coming from the banners -- this is coming from the underlying JS that loads the banners [18:31:26] mwalker: right [18:31:44] makes sense... it's the controller that allocated the banners [18:31:53] * aude can't disable that [18:32:02] aude: reedy_ just did it [18:32:06] yep [18:32:51] Why is everything being duplicated by logmsgbot in -tech too? [18:32:53] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf12 [18:32:59] Logged the message, Master [18:33:17] PROBLEM - Puppet freshness on db1049 is CRITICAL: Puppet has not run in the last 10 hours [18:33:45] I presume self reviewing is okay in emergencies? [18:33:59] xyzram: thanks [18:34:14] because logmsgbot is in tech and operations [18:34:16] when it's 'revert this' or 'turn this off' to keep the site from falling over, you bet [18:34:17] np [18:34:21] because if not that would be silly... [18:34:26] Thehelpfulone: what apergos said [18:34:30] Reedy: https://gerrit.wikimedia.org/r/#/c/55616/ [18:34:33] heh good :-) [18:34:36] plus its easy to click revert since it ties to old edit that was approved [18:34:47] so you also have the link of undoing in the audit trail [18:34:59] Reverting locally on fenari is the easiest [18:35:02] you can tidy up later [18:35:06] (granted, then you still self review your revert, but yea, thats considered normal) [18:35:08] yep :) [18:35:17] PROBLEM - Puppet freshness on mw1065 is CRITICAL: Puppet has not run in the last 10 hours [18:35:21] New patchset: Reedy; "enwiki to 1.21wmf12" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55619 [18:35:28] true ehough [18:35:30] enough even [18:35:42] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55619 [18:36:57] RECOVERY - Puppet freshness on analytics1003 is OK: puppet ran at Mon Mar 25 18:36:48 UTC 2013 [18:36:57] RECOVERY - Puppet freshness on analytics1006 is OK: puppet ran at Mon Mar 25 18:36:53 UTC 2013 [18:36:57] RECOVERY - Puppet freshness on analytics1008 is OK: puppet ran at Mon Mar 25 18:36:53 UTC 2013 [18:37:09] RECOVERY - Puppet freshness on analytics1009 is OK: puppet ran at Mon Mar 25 18:36:58 UTC 2013 [18:37:09] RECOVERY - Puppet freshness on analytics1005 is OK: puppet ran at Mon Mar 25 18:36:58 UTC 2013 [18:37:09] RECOVERY - Puppet freshness on analytics1015 is OK: puppet ran at Mon Mar 25 18:36:58 UTC 2013 [18:37:09] RECOVERY - Puppet freshness on analytics1022 is OK: puppet ran at Mon Mar 25 18:37:03 UTC 2013 [18:37:09] RECOVERY - Puppet freshness on analytics1012 is OK: puppet ran at Mon Mar 25 18:37:03 UTC 2013 [18:37:09] RECOVERY - Puppet freshness on analytics1016 is OK: puppet ran at Mon Mar 25 18:37:03 UTC 2013 [18:37:10] RECOVERY - Puppet freshness on analytics1017 is OK: puppet ran at Mon Mar 25 18:37:03 UTC 2013 [18:37:17] RECOVERY - Puppet freshness on analytics1019 is OK: puppet ran at Mon Mar 25 18:37:08 UTC 2013 [18:37:18] RECOVERY - Puppet freshness on analytics1013 is OK: puppet ran at Mon Mar 25 18:37:09 UTC 2013 [18:37:27] RECOVERY - Puppet freshness on analytics1014 is OK: puppet ran at Mon Mar 25 18:37:24 UTC 2013 [18:37:27] RECOVERY - Puppet freshness on analytics1011 is OK: puppet ran at Mon Mar 25 18:37:25 UTC 2013 [18:37:40] RECOVERY - Puppet freshness on analytics1018 is OK: puppet ran at Mon Mar 25 18:37:31 UTC 2013 [18:38:20] so; if anyone is curious; the CentralNotice problem was due to an irritating 'feature' of jQuery: http://stackoverflow.com/questions/7054795/adding-a-script-to-the-page-dynamically-with-jquery-never-uses-the-cached-file [18:38:30] yikes [18:38:42] features are the best [18:39:11] oh boy [18:39:12] I knew about that feature and I'm usually more than 2 hops from frontend code [18:39:17] ah there we go [18:39:22] paravoid: hehe [18:39:24] it's kinda baffling how you could have missed that :) [18:39:27] RECOVERY - Puppet freshness on analytics1021 is OK: puppet ran at Mon Mar 25 18:39:26 UTC 2013 [18:39:35] lol [18:40:30] paravoid: we originally had it in; but I removed it on request from mobile because they had concerns that setting this globally (which we were doing in the CN controller) would break their stuff [18:40:37] New patchset: Pyoungmeister; "re-enabling a lot of our monitoring that fall off the edge" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55620 [18:40:38] I assumed jQuery had it's own local cache [18:40:38] paravoid: can you look at that and tell me what you think? ^^ [18:40:58] notpeter: \o/ [18:41:01] paravoid: not that it would add cache breaking functionality [18:41:13] paravoid: I would like to at least have raid monitoroing again.... [18:41:17] RECOVERY - Puppet freshness on search1022 is OK: puppet ran at Mon Mar 25 18:41:11 UTC 2013 [18:41:17] RECOVERY - Puppet freshness on search1006 is OK: puppet ran at Mon Mar 25 18:41:11 UTC 2013 [18:41:17] RECOVERY - Puppet freshness on search13 is OK: puppet ran at Mon Mar 25 18:41:11 UTC 2013 [18:41:17] RECOVERY - Puppet freshness on search17 is OK: puppet ran at Mon Mar 25 18:41:11 UTC 2013 [18:41:17] RECOVERY - Puppet freshness on search1002 is OK: puppet ran at Mon Mar 25 18:41:16 UTC 2013 [18:41:18] RECOVERY - Puppet freshness on search1014 is OK: puppet ran at Mon Mar 25 18:41:16 UTC 2013 [18:41:18] RECOVERY - Puppet freshness on search1007 is OK: puppet ran at Mon Mar 25 18:41:16 UTC 2013 [18:41:19] turns out, that's pretty useful [18:41:27] RECOVERY - Puppet freshness on search1003 is OK: puppet ran at Mon Mar 25 18:41:23 UTC 2013 [18:41:49] RECOVERY - Puppet freshness on search1023 is OK: puppet ran at Mon Mar 25 18:41:40 UTC 2013 [18:41:49] RECOVERY - Puppet freshness on search1012 is OK: puppet ran at Mon Mar 25 18:41:46 UTC 2013 [18:41:49] RECOVERY - Puppet freshness on search1013 is OK: puppet ran at Mon Mar 25 18:41:46 UTC 2013 [18:41:49] notpeter: argh with the duplication between files/icinga and files/nagios [18:42:04] LeslieCarr: you've been doing most of those manifests, can we please merge icinga/nagios again? [18:42:07] RECOVERY - Puppet freshness on search1016 is OK: puppet ran at Mon Mar 25 18:42:03 UTC 2013 [18:42:07] RECOVERY - Puppet freshness on search1011 is OK: puppet ran at Mon Mar 25 18:42:03 UTC 2013 [18:42:10] we've been duplicating plugins all over [18:42:17] and then updating them and forgetting half of the changes [18:42:21] paravoid: yeah.... [18:42:37] yes, paravoid this afternoon i'll do another round of de-duping [18:42:37] RECOVERY - Puppet freshness on search1024 is OK: puppet ran at Mon Mar 25 18:42:33 UTC 2013 [18:42:37] RECOVERY - Puppet freshness on search1019 is OK: puppet ran at Mon Mar 25 18:42:33 UTC 2013 [18:42:37] RECOVERY - Puppet freshness on search1020 is OK: puppet ran at Mon Mar 25 18:42:34 UTC 2013 [18:42:37] RECOVERY - Puppet freshness on search1005 is OK: puppet ran at Mon Mar 25 18:42:34 UTC 2013 [18:42:37] RECOVERY - Puppet freshness on search1017 is OK: puppet ran at Mon Mar 25 18:42:34 UTC 2013 [18:42:47] RECOVERY - Puppet freshness on search1009 is OK: puppet ran at Mon Mar 25 18:42:39 UTC 2013 [18:43:07] RECOVERY - Puppet freshness on search19 is OK: puppet ran at Mon Mar 25 18:43:01 UTC 2013 [18:43:13] notpeter: case in point, you updated nrpe_local only for nagios, not icinga [18:43:26] notpeter: templates/nagios/nrpe_local.cfg.erb vs. templates/icinga/nrpe_local.cfg.erb [18:43:36] paravoid: doh. I was trying to use the define to move away from that file all together [18:43:46] I noticed [18:43:47] RECOVERY - Puppet freshness on search14 is OK: puppet ran at Mon Mar 25 18:43:41 UTC 2013 [18:43:54] but the lines are in the icinga file too [18:44:07] RECOVERY - Puppet freshness on search1004 is OK: puppet ran at Mon Mar 25 18:43:58 UTC 2013 [18:44:12] paravoid: yep [18:44:14] patching now [18:44:21] thank you for reviewing :) [18:44:32] check_dpkg could use some love too [18:44:38] echo "This plugin checks hardware status using the lm_sensors package." [18:44:42] lol :) [18:44:45] New patchset: Pyoungmeister; "re-enabling a lot of our monitoring that fall off the edge" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55620 [18:44:50] paravoid: well... I'm trying to at least get us back to some checks existing [18:44:55] I really just care about the raid, tbh [18:45:01] cuz, man.... that's something we should know [18:45:03] for srs [18:45:41] heh [18:45:52] I'm actually a bit scared as to what we'll find with it... [18:46:07] !log reedy synchronized php-1.21wmf12/extensions/Wikibase [18:46:12] haha [18:46:14] Logged the message, Master [18:46:24] but, that's something we need to know [18:46:37] ignorance is bliss [18:47:19] Can someone run sync-common as root on mw1209? loads of "rsync: failed to set times on" [18:47:50] s/bliss/a heart attack risk/ [18:48:19] paravoid: ok, ithink I'm going to merge [18:48:22] binasher: are you seeing any other massive amounts of queries with _= right now? [18:48:26] and if anything still isn't working, I'll fix [18:49:10] notpeter: Can you restart the incrementalUpdater for now ? [18:49:23] mwalker: still seeing some for Special:BannerRandom [18:49:25] also [18:49:27] http://commons.wikimedia.org/w/api.php?callback=jQuery183041549299135330253_1364236704727&action=parse&page=File%3AVladimir_Putin_singing_Blueberry_Hill.ogv&smaxage=3600&maxage=3600&format=json&_=1364237326732 [18:49:33] xyzram: yeah [18:50:00] xyzram: done [18:50:15] thanks. [18:50:33] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55620 [18:51:15] cmjohnson1: I'm probably going to make a lot of "replace disk in X" tickets in the next day.... [18:51:19] just to warn you [18:51:45] notpeter: a bunch of failures? [18:52:15] cmjohnson1: after a bunch of actually testing for raid fails ;) [18:53:01] okay...i will look for the tickets [18:53:54] New patchset: Aude; "Fix Wikibase settings, allow per wiki overrides for data inclusion" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55621 [18:54:29] Reedy: test2 will work better with https://gerrit.wikimedia.org/r/#/c/55621/ [18:54:44] * aude has to put the common settings above where we include the per wiki settings [18:55:06] New patchset: Dzahn; "add favicon/gpl copy/logo/ small changes to CSS / api/index/rank.php sync live hacks to package repo" [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/55622 [18:55:22] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/55621 [18:56:47] !log reedy synchronized wmf-config/CommonSettings.php [18:56:53] Logged the message, Master [18:57:08] RECOVERY - Puppet freshness on db1027 is OK: puppet ran at Mon Mar 25 18:57:05 UTC 2013 [18:57:09] RECOVERY - Puppet freshness on db1026 is OK: puppet ran at Mon Mar 25 18:57:06 UTC 2013 [18:57:17] RECOVERY - Puppet freshness on db1028 is OK: puppet ran at Mon Mar 25 18:57:11 UTC 2013 [18:57:29] RECOVERY - Puppet freshness on db1010 is OK: puppet ran at Mon Mar 25 18:57:26 UTC 2013 [18:57:39] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Mon Mar 25 18:57:29 UTC 2013 [18:57:48] RECOVERY - Puppet freshness on db1011 is OK: puppet ran at Mon Mar 25 18:57:40 UTC 2013 [18:57:57] RECOVERY - Puppet freshness on db1050 is OK: puppet ran at Mon Mar 25 18:57:55 UTC 2013 [18:58:01] test2 looks good now [18:58:57] RECOVERY - Puppet freshness on db1051 is OK: puppet ran at Mon Mar 25 18:58:50 UTC 2013 [19:00:37] New patchset: Pyoungmeister; "/usr/bin/local != /usr/local/bin" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55623 [19:02:05] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55623 [19:02:06] Change merged: Dzahn; [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/55622 [19:02:17] RECOVERY - Puppet freshness on db29 is OK: puppet ran at Mon Mar 25 19:02:16 UTC 2013 [19:02:57] New review: Dzahn; "needs to wait until wikivoyage table actually exists" [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/53928 [19:02:58] Reedy: we have a new special page on wikidata [19:03:06] this means we need localisation cache updated [19:03:06] http://wikidata.org/wiki/Special:DispatchStats [19:03:54] I think it looks fine like that :p [19:04:38] heh [19:04:45] hopefully the stats improve also [19:05:22] the dispatcher prioritizes and handles changes differently/better now [19:05:48] and we can have a clue what's happening without poking you by looking at the special page [19:07:17] RECOVERY - Puppet freshness on db59 is OK: puppet ran at Mon Mar 25 19:07:14 UTC 2013 [19:11:27] RECOVERY - Puppet freshness on es10 is OK: puppet ran at Mon Mar 25 19:11:26 UTC 2013 [19:11:27] RECOVERY - Puppet freshness on es6 is OK: puppet ran at Mon Mar 25 19:11:26 UTC 2013 [19:11:27] RECOVERY - Puppet freshness on es7 is OK: puppet ran at Mon Mar 25 19:11:26 UTC 2013 [19:11:40] RECOVERY - Puppet freshness on es9 is OK: puppet ran at Mon Mar 25 19:11:31 UTC 2013 [19:11:40] RECOVERY - Puppet freshness on es5 is OK: puppet ran at Mon Mar 25 19:11:31 UTC 2013 [19:11:40] RECOVERY - Puppet freshness on es1009 is OK: puppet ran at Mon Mar 25 19:11:31 UTC 2013 [19:11:40] RECOVERY - Puppet freshness on es1006 is OK: puppet ran at Mon Mar 25 19:11:31 UTC 2013 [19:15:22] New patchset: Faidon; "Add httpry to base::standard-packages" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55624 [19:16:57] PROBLEM - mysqld processes on db67 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [19:17:57] RECOVERY - Puppet freshness on db67 is OK: puppet ran at Mon Mar 25 19:17:53 UTC 2013 [19:18:32] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55624 [19:20:09] New patchset: Asher; "db67 -> mariadb" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55625 [19:20:32] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/55625 [19:24:57] RECOVERY - mysqld processes on db67 is OK: PROCS OK: 1 process with command name mysqld [19:25:51] binasher: I'm going to put the jquery caching default back into CentralNotice -- but do you have a record of all the urls (that are not centralnotice) that did the _= thing? [19:26:01] or is site load totally fine right now; and I don't need to re add in the caching default [19:26:19] and we can fix all the bad calls as we find them [19:29:50] mwalker: i think the non cn calls are ok, in that they're on requests that wouldn't be cached anyways [19:29:59] cool [19:30:15] but if you look at some of the examples, like - http://en.wikipedia.org/w/api.php?callback=jQuery17203923157136636861_1364236130788&action=query&list=random&rnnamespace=0&rnlimit=1&redirects=1&format=json&_=1364239605315 [19:30:45] the json response from mediawiki includes {"warnings":{"main":{"*":"Unrecognized parameter: '_'"}} [19:30:50] heh -- I love how that breaks the API... Krinkle ^ [19:31:20] mediawiki generally does the right thing wrt squid/varnish caching by setting appropriate headers [19:31:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:31:56] so i think that jquery behavior should be disabled across the board [19:32:26] mwalker: Hm.. I don't know what the url is coming from, but this isn't from