[00:03:16] (03PS1) 10Mwalker: Enable CentralNotice CrossWiki Hiding [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92817 [00:04:37] yurik_: MaxSem : what paravoid said: +1 to a post-mortem, plz [00:04:53] shoor [00:05:01] what ori-l said :) [00:05:33] whoever :) [00:05:45] yurik_: MaxSem , send to engineering@ [00:05:49] greg-g: technically its not MaxSem at all [00:06:02] greg-g, why not wikitech? [00:06:06] postmortems aren't to assign blame [00:06:33] in theory :) [00:06:55] right, just figure out what happened, he helped diagnose and such [00:07:00] AaronSchulz: shush :) [00:07:31] MaxSem: mostly that it's just deploy specific and where they've gone before... I don't have a policy against wikitech [00:08:20] (unless it includes private info, of course) [00:08:21] my point is that engineering stuff should be on wikitech and that volunteers might be interested, too [00:08:29] sure [00:08:54] but, cc engineering@ since not all wmf engineers (and thus deployers) read all of wikitech-l :) [00:14:24] We have a blame wheel. [00:14:27] Let's have a look. [00:14:47] Domas. [00:17:04] sounds legit [00:17:22] We lost WP:BLAME. [00:17:26] To WikiBlame. [00:17:33] But we got WP:BLAMEWHEEL. [00:17:43] https://en.wikipedia.org/wiki/Wikipedia:BLAMEWHEEL [00:18:58] (03PS1) 10Dr0ptp4kt: Further constrain W0 X-CS setting to mobile Wikipedia, for now. [operations/puppet] - 10https://gerrit.wikimedia.org/r/92818 [00:19:11] greg-g, MaxSem so whole should i send it to? [00:19:57] yurik_: engineering@ and wikitech-l (I can't think of anything you might have said that isn't ok to share publicly) [00:20:44] yurik_, I'm already writing it [00:20:54] MaxSem: i thought i was :) [00:21:09] since its zero, not MF [00:21:13] okay [00:21:17] up to you really [00:21:29] (03CR) 10Faidon Liambotis: [C: 04-1] "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/92818 (owner: 10Dr0ptp4kt) [00:21:34] draft discarded:P [00:21:58] heh [00:22:08] MaxSem: https://etherpad.wikimedia.org/p/zerocrash [00:22:10] you can both write it and see who gets the most upboats? [00:22:11] (03CR) 10Dr0ptp4kt: "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/92818 (owner: 10Dr0ptp4kt) [00:22:49] gnite [00:22:58] that's a good handle and/or sci fi novel name :) [00:23:02] g'night paravoid [00:23:29] greg-g: LD is 4pm in each timezone, right? [00:23:51] ori-l: ....... [00:23:52] (03PS2) 10Dr0ptp4kt: Further constrain W0 X-CS setting to mobile Wikipedia, for now. [operations/puppet] - 10https://gerrit.wikimedia.org/r/92818 [00:24:22] btw, I'm tethering, and in and out of connectivity, so if I don't answer just $#*@(!!@#$ CARRIER LOST [00:24:24] (03CR) 10Dr0ptp4kt: "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/92818 (owner: 10Dr0ptp4kt) [00:39:32] MaxSem: is that you typing in etherpad? [00:39:40] in pink [00:39:45] yup [00:39:59] MaxSem: ok, are we done? [00:40:48] I'd like more copyediting [00:41:10] do we have an english major around? [00:42:07] it wouldn't surprise me [00:42:24] * greg-g looks, but I'm slow, tethering on 3g-ish [00:44:15] 1 [00:44:15] An error occured while loading the pad [00:44:15] Error: Permission denied to access property 'valueOf' in https://etherpad.wikimedia.org/p/zerocrash (line 514) [00:44:18] 0 [00:44:41] WFM [00:45:33] probably just tethering issues [00:46:32] !log running MONITOR for 60 seconds on redis instance on mc1002 [00:46:47] Logged the message, Master [00:50:48] yurik_, how does it look now? [00:51:04] MaxSem: yurik_ looks good to me, who cares about grammemmer [00:51:38] yurik_, fire away [00:52:04] looks good [00:52:06] sending [00:53:51] sent [00:54:10] we can sleep now [00:54:17] oh, max wanted to send it to public [01:01:38] 23:03pm UTC, heh. [01:40:02] (03PS3) 10Reedy: Remove defining of MEDIAWIKI constant, done in WebStart.php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92800 [01:40:07] (03CR) 10Reedy: [C: 032] Remove defining of MEDIAWIKI constant, done in WebStart.php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92800 (owner: 10Reedy) [01:40:23] (03Merged) 10jenkins-bot: Remove defining of MEDIAWIKI constant, done in WebStart.php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92800 (owner: 10Reedy) [01:42:42] (03PS1) 10Reedy: Wrap extrac2.php in ob_start()/ob_flush() [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92828 [02:12:24] (03PS2) 10MZMcBride: Wrap extrac2.php in ob_start()/ob_flush() [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92828 (owner: 10Reedy) [02:13:01] (03CR) 10MZMcBride: "You may want to note in the file why you've added these lines. Though I suppose git blame covers that." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/92828 (owner: 10Reedy) [02:25:43] !log LocalisationUpdate completed (1.23wmf1) at Thu Oct 31 02:25:43 UTC 2013 [02:26:01] Logged the message, Master [02:47:38] !log LocalisationUpdate completed (1.22wmf22) at Thu Oct 31 02:47:38 UTC 2013 [02:47:54] Logged the message, Master [03:03:24] PROBLEM - MySQL Replication Heartbeat on db57 is CRITICAL: CRIT replication delay 306 seconds [03:04:04] PROBLEM - MySQL Slave Delay on db57 is CRITICAL: CRIT replication delay 326 seconds [03:07:06] RECOVERY - MySQL Slave Delay on db57 is OK: OK replication delay 107 seconds [03:07:26] RECOVERY - MySQL Replication Heartbeat on db57 is OK: OK replication delay 40 seconds [03:26:30] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Oct 31 03:26:30 UTC 2013 [03:26:46] Logged the message, Master [04:29:52] RECOVERY - check_job_queue on hume is OK: JOBQUEUE OK - all job queues below 10,000 [04:33:02] PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:35:02] RECOVERY - check_job_queue on hume is OK: JOBQUEUE OK - all job queues below 10,000 [04:38:02] PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:59:32] PROBLEM - RAID on arsenic is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:00:22] RECOVERY - RAID on arsenic is OK: OK: no RAID installed [05:05:35] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:06:35] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 1 logical drive(s), 4 physical drive(s) [05:07:45] PROBLEM - DPKG on arsenic is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:08:45] RECOVERY - DPKG on arsenic is OK: All packages OK [05:11:35] PROBLEM - RAID on arsenic is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:12:35] RECOVERY - RAID on arsenic is OK: OK: no RAID installed [06:58:51] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: No successful Puppet run in the last 10 hours [07:17:34] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [07:18:34] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [07:31:24] PROBLEM - Puppet freshness on tmh1 is CRITICAL: No successful Puppet run in the last 10 hours [07:46:24] PROBLEM - Puppet freshness on mw10 is CRITICAL: No successful Puppet run in the last 10 hours [07:46:24] PROBLEM - Puppet freshness on mw1011 is CRITICAL: No successful Puppet run in the last 10 hours [07:46:24] PROBLEM - Puppet freshness on mw1054 is CRITICAL: No successful Puppet run in the last 10 hours [07:46:24] PROBLEM - Puppet freshness on mw1211 is CRITICAL: No successful Puppet run in the last 10 hours [07:46:24] PROBLEM - Puppet freshness on mw1129 is CRITICAL: No successful Puppet run in the last 10 hours [07:46:24] PROBLEM - Puppet freshness on mw1213 is CRITICAL: No successful Puppet run in the last 10 hours [07:46:24] PROBLEM - Puppet freshness on mw23 is CRITICAL: No successful Puppet run in the last 10 hours [07:46:25] PROBLEM - Puppet freshness on mw60 is CRITICAL: No successful Puppet run in the last 10 hours [07:47:24] PROBLEM - Puppet freshness on antimony is CRITICAL: No successful Puppet run in the last 10 hours [07:47:24] PROBLEM - Puppet freshness on mw1206 is CRITICAL: No successful Puppet run in the last 10 hours [07:47:24] PROBLEM - Puppet freshness on labstore3 is CRITICAL: No successful Puppet run in the last 10 hours [07:47:24] PROBLEM - Puppet freshness on mw42 is CRITICAL: No successful Puppet run in the last 10 hours [07:47:24] PROBLEM - Puppet freshness on srv242 is CRITICAL: No successful Puppet run in the last 10 hours [07:47:24] PROBLEM - Puppet freshness on mw1208 is CRITICAL: No successful Puppet run in the last 10 hours [07:47:24] PROBLEM - Puppet freshness on mw55 is CRITICAL: No successful Puppet run in the last 10 hours [07:47:25] PROBLEM - Puppet freshness on mw75 is CRITICAL: No successful Puppet run in the last 10 hours [07:48:24] PROBLEM - Puppet freshness on hume is CRITICAL: No successful Puppet run in the last 10 hours [07:48:24] PROBLEM - Puppet freshness on mw1049 is CRITICAL: No successful Puppet run in the last 10 hours [07:48:24] PROBLEM - Puppet freshness on mw1084 is CRITICAL: No successful Puppet run in the last 10 hours [07:48:24] PROBLEM - Puppet freshness on mw116 is CRITICAL: No successful Puppet run in the last 10 hours [07:48:24] PROBLEM - Puppet freshness on mw1180 is CRITICAL: No successful Puppet run in the last 10 hours [07:48:24] PROBLEM - Puppet freshness on mw13 is CRITICAL: No successful Puppet run in the last 10 hours [07:48:24] PROBLEM - Puppet freshness on mw74 is CRITICAL: No successful Puppet run in the last 10 hours [07:48:25] PROBLEM - Puppet freshness on snapshot1002 is CRITICAL: No successful Puppet run in the last 10 hours [07:48:25] PROBLEM - Puppet freshness on sodium is CRITICAL: No successful Puppet run in the last 10 hours [07:48:26] PROBLEM - Puppet freshness on srv299 is CRITICAL: No successful Puppet run in the last 10 hours [07:49:24] PROBLEM - Puppet freshness on mw1030 is CRITICAL: No successful Puppet run in the last 10 hours [07:49:24] PROBLEM - Puppet freshness on mw1165 is CRITICAL: No successful Puppet run in the last 10 hours [07:49:24] PROBLEM - Puppet freshness on mw1181 is CRITICAL: No successful Puppet run in the last 10 hours [07:49:24] PROBLEM - Puppet freshness on srv291 is CRITICAL: No successful Puppet run in the last 10 hours [07:50:24] PROBLEM - Puppet freshness on mw1053 is CRITICAL: No successful Puppet run in the last 10 hours [07:50:24] PROBLEM - Puppet freshness on mw39 is CRITICAL: No successful Puppet run in the last 10 hours [07:50:24] PROBLEM - Puppet freshness on mw1198 is CRITICAL: No successful Puppet run in the last 10 hours [07:50:24] PROBLEM - Puppet freshness on mw1116 is CRITICAL: No successful Puppet run in the last 10 hours [07:50:24] PROBLEM - Puppet freshness on mw1074 is CRITICAL: No successful Puppet run in the last 10 hours [07:50:24] PROBLEM - Puppet freshness on srv272 is CRITICAL: No successful Puppet run in the last 10 hours [07:51:24] PROBLEM - Puppet freshness on brewster is CRITICAL: No successful Puppet run in the last 10 hours [07:51:24] PROBLEM - Puppet freshness on mw33 is CRITICAL: No successful Puppet run in the last 10 hours [07:51:24] PROBLEM - Puppet freshness on mw64 is CRITICAL: No successful Puppet run in the last 10 hours [07:52:24] PROBLEM - Puppet freshness on mw1001 is CRITICAL: No successful Puppet run in the last 10 hours [07:52:24] PROBLEM - Puppet freshness on mw53 is CRITICAL: No successful Puppet run in the last 10 hours [07:52:24] PROBLEM - Puppet freshness on mw1185 is CRITICAL: No successful Puppet run in the last 10 hours [07:52:24] PROBLEM - Puppet freshness on mw70 is CRITICAL: No successful Puppet run in the last 10 hours [07:52:24] PROBLEM - Puppet freshness on mw1022 is CRITICAL: No successful Puppet run in the last 10 hours [07:53:24] PROBLEM - Puppet freshness on mw111 is CRITICAL: No successful Puppet run in the last 10 hours [07:53:24] PROBLEM - Puppet freshness on mw87 is CRITICAL: No successful Puppet run in the last 10 hours [07:53:24] PROBLEM - Puppet freshness on srv285 is CRITICAL: No successful Puppet run in the last 10 hours [07:54:24] PROBLEM - Puppet freshness on mw1086 is CRITICAL: No successful Puppet run in the last 10 hours [07:54:24] PROBLEM - Puppet freshness on mw1093 is CRITICAL: No successful Puppet run in the last 10 hours [07:54:24] PROBLEM - Puppet freshness on mw1112 is CRITICAL: No successful Puppet run in the last 10 hours [07:54:24] PROBLEM - Puppet freshness on mw1139 is CRITICAL: No successful Puppet run in the last 10 hours [07:54:24] PROBLEM - Puppet freshness on mw1090 is CRITICAL: No successful Puppet run in the last 10 hours [07:54:24] PROBLEM - Puppet freshness on mw115 is CRITICAL: No successful Puppet run in the last 10 hours [07:54:24] PROBLEM - Puppet freshness on mw1215 is CRITICAL: No successful Puppet run in the last 10 hours [07:54:25] PROBLEM - Puppet freshness on srv280 is CRITICAL: No successful Puppet run in the last 10 hours [07:54:25] PROBLEM - Puppet freshness on mw1219 is CRITICAL: No successful Puppet run in the last 10 hours [07:54:26] PROBLEM - Puppet freshness on mw1203 is CRITICAL: No successful Puppet run in the last 10 hours [07:54:26] PROBLEM - Puppet freshness on mw1220 is CRITICAL: No successful Puppet run in the last 10 hours [07:54:27] PROBLEM - Puppet freshness on terbium is CRITICAL: No successful Puppet run in the last 10 hours [07:55:24] PROBLEM - Puppet freshness on mw1110 is CRITICAL: No successful Puppet run in the last 10 hours [07:55:24] PROBLEM - Puppet freshness on mw113 is CRITICAL: No successful Puppet run in the last 10 hours [07:55:24] PROBLEM - Puppet freshness on mw1135 is CRITICAL: No successful Puppet run in the last 10 hours [07:55:24] PROBLEM - Puppet freshness on mw1158 is CRITICAL: No successful Puppet run in the last 10 hours [07:55:24] PROBLEM - Puppet freshness on mw92 is CRITICAL: No successful Puppet run in the last 10 hours [07:55:24] PROBLEM - Puppet freshness on srv298 is CRITICAL: No successful Puppet run in the last 10 hours [07:56:24] PROBLEM - Puppet freshness on mw1073 is CRITICAL: No successful Puppet run in the last 10 hours [07:56:24] PROBLEM - Puppet freshness on mw25 is CRITICAL: No successful Puppet run in the last 10 hours [07:56:24] PROBLEM - Puppet freshness on mw1199 is CRITICAL: No successful Puppet run in the last 10 hours [07:56:24] PROBLEM - Puppet freshness on srv293 is CRITICAL: No successful Puppet run in the last 10 hours [07:56:24] PROBLEM - Puppet freshness on srv251 is CRITICAL: No successful Puppet run in the last 10 hours [07:57:21] err: Failed to apply catalog: Could not find dependency Systemuser[pybal-check] for File[/var/lib/pybal-check/.ssh] at /etc/puppet/modules/applicationserver/manifests/pybal_check.pp:20 [07:57:24] PROBLEM - Puppet freshness on mw1019 is CRITICAL: No successful Puppet run in the last 10 hours [07:57:24] PROBLEM - Puppet freshness on mw1058 is CRITICAL: No successful Puppet run in the last 10 hours [07:57:24] PROBLEM - Puppet freshness on mw1070 is CRITICAL: No successful Puppet run in the last 10 hours [07:57:24] PROBLEM - Puppet freshness on mw1085 is CRITICAL: No successful Puppet run in the last 10 hours [07:57:24] PROBLEM - Puppet freshness on mw12 is CRITICAL: No successful Puppet run in the last 10 hours [07:57:24] PROBLEM - Puppet freshness on snapshot3 is CRITICAL: No successful Puppet run in the last 10 hours [07:57:24] PROBLEM - Puppet freshness on mw1157 is CRITICAL: No successful Puppet run in the last 10 hours [07:57:25] PROBLEM - Puppet freshness on srv235 is CRITICAL: No successful Puppet run in the last 10 hours [07:57:25] PROBLEM - Puppet freshness on srv274 is CRITICAL: No successful Puppet run in the last 10 hours [07:57:26] PROBLEM - Puppet freshness on srv268 is CRITICAL: No successful Puppet run in the last 10 hours [07:58:24] PROBLEM - Puppet freshness on mw1102 is CRITICAL: No successful Puppet run in the last 10 hours [07:58:24] PROBLEM - Puppet freshness on mw1184 is CRITICAL: No successful Puppet run in the last 10 hours [07:58:24] PROBLEM - Puppet freshness on mw1191 is CRITICAL: No successful Puppet run in the last 10 hours [07:58:24] PROBLEM - Puppet freshness on srv239 is CRITICAL: No successful Puppet run in the last 10 hours [07:58:24] PROBLEM - Puppet freshness on mw96 is CRITICAL: No successful Puppet run in the last 10 hours [07:59:24] PROBLEM - Puppet freshness on mw1035 is CRITICAL: No successful Puppet run in the last 10 hours [07:59:24] PROBLEM - Puppet freshness on mw1013 is CRITICAL: No successful Puppet run in the last 10 hours [07:59:24] PROBLEM - Puppet freshness on mw1036 is CRITICAL: No successful Puppet run in the last 10 hours [07:59:24] PROBLEM - Puppet freshness on mw63 is CRITICAL: No successful Puppet run in the last 10 hours [07:59:24] PROBLEM - Puppet freshness on mw69 is CRITICAL: No successful Puppet run in the last 10 hours [07:59:24] PROBLEM - Puppet freshness on mw1096 is CRITICAL: No successful Puppet run in the last 10 hours [07:59:24] PROBLEM - Puppet freshness on searchidx2 is CRITICAL: No successful Puppet run in the last 10 hours [07:59:25] PROBLEM - Puppet freshness on srv254 is CRITICAL: No successful Puppet run in the last 10 hours [07:59:25] PROBLEM - Puppet freshness on mw71 is CRITICAL: No successful Puppet run in the last 10 hours [07:59:26] PROBLEM - Puppet freshness on tmh1002 is CRITICAL: No successful Puppet run in the last 10 hours [07:59:43] (03CR) 10Ori.livneh: "err: Failed to apply catalog: Could not find dependency Systemuser[pybal-check] for File[/var/lib/pybal-check/.ssh] at /etc/puppet/modules" [operations/puppet] - 10https://gerrit.wikimedia.org/r/92339 (owner: 10Andrew Bogott) [08:00:24] PROBLEM - Puppet freshness on mw1192 is CRITICAL: No successful Puppet run in the last 10 hours [08:00:24] PROBLEM - Puppet freshness on mw1216 is CRITICAL: No successful Puppet run in the last 10 hours [08:00:24] PROBLEM - Puppet freshness on mw32 is CRITICAL: No successful Puppet run in the last 10 hours [08:00:24] PROBLEM - Puppet freshness on mw86 is CRITICAL: No successful Puppet run in the last 10 hours [08:00:24] PROBLEM - Puppet freshness on mw18 is CRITICAL: No successful Puppet run in the last 10 hours [08:01:24] PROBLEM - Puppet freshness on mw1005 is CRITICAL: No successful Puppet run in the last 10 hours [08:01:24] PROBLEM - Puppet freshness on mw1067 is CRITICAL: No successful Puppet run in the last 10 hours [08:01:24] PROBLEM - Puppet freshness on mw1028 is CRITICAL: No successful Puppet run in the last 10 hours [08:01:24] PROBLEM - Puppet freshness on mw1109 is CRITICAL: No successful Puppet run in the last 10 hours [08:02:24] PROBLEM - Puppet freshness on mw1006 is CRITICAL: No successful Puppet run in the last 10 hours [08:02:24] PROBLEM - Puppet freshness on mw31 is CRITICAL: No successful Puppet run in the last 10 hours [08:02:24] PROBLEM - Puppet freshness on snapshot2 is CRITICAL: No successful Puppet run in the last 10 hours [08:02:24] PROBLEM - Puppet freshness on mw1012 is CRITICAL: No successful Puppet run in the last 10 hours [08:02:24] PROBLEM - Puppet freshness on mw73 is CRITICAL: No successful Puppet run in the last 10 hours [08:02:24] PROBLEM - Puppet freshness on srv248 is CRITICAL: No successful Puppet run in the last 10 hours [08:02:24] PROBLEM - Puppet freshness on srv265 is CRITICAL: No successful Puppet run in the last 10 hours [08:02:25] PROBLEM - Puppet freshness on srv277 is CRITICAL: No successful Puppet run in the last 10 hours [08:02:28] (03PS1) 10Ori.livneh: Systemuser -> Generic::Systemuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/92838 [08:02:52] yep, there's more of them too [08:03:20] there's twemproxy upstart, [08:03:24] PROBLEM - Puppet freshness on mw101 is CRITICAL: No successful Puppet run in the last 10 hours [08:03:24] PROBLEM - Puppet freshness on mw1160 is CRITICAL: No successful Puppet run in the last 10 hours [08:03:24] PROBLEM - Puppet freshness on mw103 is CRITICAL: No successful Puppet run in the last 10 hours [08:03:24] PROBLEM - Puppet freshness on mw1187 is CRITICAL: No successful Puppet run in the last 10 hours [08:03:24] PROBLEM - Puppet freshness on mw77 is CRITICAL: No successful Puppet run in the last 10 hours [08:03:24] PROBLEM - Puppet freshness on mw54 is CRITICAL: No successful Puppet run in the last 10 hours [08:03:24] PROBLEM - Puppet freshness on mw94 is CRITICAL: No successful Puppet run in the last 10 hours [08:03:25] PROBLEM - Puppet freshness on snapshot1003 is CRITICAL: No successful Puppet run in the last 10 hours [08:03:25] PROBLEM - Puppet freshness on srv295 is CRITICAL: No successful Puppet run in the last 10 hours [08:03:45] oh yeah [08:03:47] i'll update the patch [08:03:54] thank you [08:04:20] I should replace twemproxy with my packages at some point [08:04:24] PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours [08:04:24] PROBLEM - Puppet freshness on mw51 is CRITICAL: No successful Puppet run in the last 10 hours [08:04:24] PROBLEM - Puppet freshness on srv287 is CRITICAL: No successful Puppet run in the last 10 hours [08:04:24] PROBLEM - Puppet freshness on tmh2 is CRITICAL: No successful Puppet run in the last 10 hours [08:04:26] I'll remind the author to test puppet on a few hosts for changesets of this magnitude [08:04:30] I was waiting for a Debian upload but upstream wasn't very friendly [08:06:12] bummer [08:06:24] PROBLEM - Puppet freshness on mw1002 is CRITICAL: No successful Puppet run in the last 10 hours [08:06:24] PROBLEM - Puppet freshness on mw1042 is CRITICAL: No successful Puppet run in the last 10 hours [08:06:24] PROBLEM - Puppet freshness on mw1052 is CRITICAL: No successful Puppet run in the last 10 hours [08:06:24] PROBLEM - Puppet freshness on mw1065 is CRITICAL: No successful Puppet run in the last 10 hours [08:06:24] PROBLEM - Puppet freshness on mw1099 is CRITICAL: No successful Puppet run in the last 10 hours [08:06:32] yeah thanks icinga-wm [08:08:05] (03PS2) 10Ori.livneh: Systemuser -> Generic::Systemuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/92838 [08:08:24] PROBLEM - Puppet freshness on mw1111 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:24] PROBLEM - Puppet freshness on mw1125 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:24] PROBLEM - Puppet freshness on mw119 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:24] PROBLEM - Puppet freshness on mw1190 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:24] PROBLEM - Puppet freshness on mw38 is CRITICAL: No successful Puppet run in the last 10 hours [08:08:25] PROBLEM - Puppet freshness on tin is CRITICAL: No successful Puppet run in the last 10 hours [08:09:24] PROBLEM - Puppet freshness on mw1004 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:24] PROBLEM - Puppet freshness on mw1056 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:24] PROBLEM - Puppet freshness on mw1146 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:24] PROBLEM - Puppet freshness on mw30 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:24] PROBLEM - Puppet freshness on srv246 is CRITICAL: No successful Puppet run in the last 10 hours [08:09:25] PROBLEM - Puppet freshness on srv249 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:01] * paravoid hates systemuser [08:10:07] i know, heheh [08:10:22] but i hate upstart_service more :P [08:10:24] PROBLEM - Puppet freshness on mw1057 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:24] PROBLEM - Puppet freshness on mw1081 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:24] PROBLEM - Puppet freshness on mw1087 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:24] PROBLEM - Puppet freshness on mw1159 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:24] PROBLEM - Puppet freshness on mw1171 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:25] PROBLEM - Puppet freshness on mw1188 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:25] PROBLEM - Puppet freshness on mw1210 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:26] PROBLEM - Puppet freshness on mw58 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:26] PROBLEM - Puppet freshness on mw8 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:27] PROBLEM - Puppet freshness on snapshot1 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:27] PROBLEM - Puppet freshness on srv276 is CRITICAL: No successful Puppet run in the last 10 hours [08:10:34] RECOVERY - RAID on arsenic is OK: OK: no RAID installed [08:11:09] apergos: does that look ok? [08:11:24] PROBLEM - Puppet freshness on mw1032 is CRITICAL: No successful Puppet run in the last 10 hours [08:11:24] PROBLEM - Puppet freshness on mw57 is CRITICAL: No successful Puppet run in the last 10 hours [08:11:24] PROBLEM - Puppet freshness on srv255 is CRITICAL: No successful Puppet run in the last 10 hours [08:11:24] PROBLEM - Puppet freshness on srv273 is CRITICAL: No successful Puppet run in the last 10 hours [08:11:58] I don't know if it's all of em but let's get it out there and test on a host, see what's left [08:12:24] PROBLEM - Puppet freshness on mw1024 is CRITICAL: No successful Puppet run in the last 10 hours [08:12:24] PROBLEM - Puppet freshness on mw1033 is CRITICAL: No successful Puppet run in the last 10 hours [08:12:24] PROBLEM - Puppet freshness on mw106 is CRITICAL: No successful Puppet run in the last 10 hours [08:12:24] PROBLEM - Puppet freshness on mw1122 is CRITICAL: No successful Puppet run in the last 10 hours [08:12:24] PROBLEM - Puppet freshness on mw1201 is CRITICAL: No successful Puppet run in the last 10 hours [08:12:25] PROBLEM - Puppet freshness on mw2 is CRITICAL: No successful Puppet run in the last 10 hours [08:12:25] PROBLEM - Puppet freshness on mw35 is CRITICAL: No successful Puppet run in the last 10 hours [08:12:26] PROBLEM - Puppet freshness on mw98 is CRITICAL: No successful Puppet run in the last 10 hours [08:13:24] PROBLEM - Puppet freshness on mw1010 is CRITICAL: No successful Puppet run in the last 10 hours [08:13:24] PROBLEM - Puppet freshness on mw1091 is CRITICAL: No successful Puppet run in the last 10 hours [08:13:24] PROBLEM - Puppet freshness on mw112 is CRITICAL: No successful Puppet run in the last 10 hours [08:13:24] PROBLEM - Puppet freshness on srv262 is CRITICAL: No successful Puppet run in the last 10 hours [08:13:24] PROBLEM - Puppet freshness on srv267 is CRITICAL: No successful Puppet run in the last 10 hours [08:13:25] PROBLEM - Puppet freshness on mw1142 is CRITICAL: No successful Puppet run in the last 10 hours [08:13:25] PROBLEM - Puppet freshness on srv269 is CRITICAL: No successful Puppet run in the last 10 hours [08:14:05] (03CR) 10Ori.livneh: [C: 032] Systemuser -> Generic::Systemuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/92838 (owner: 10Ori.livneh) [08:14:24] PROBLEM - Puppet freshness on mw1027 is CRITICAL: No successful Puppet run in the last 10 hours [08:14:24] PROBLEM - Puppet freshness on mw1066 is CRITICAL: No successful Puppet run in the last 10 hours [08:14:24] PROBLEM - Puppet freshness on mw1107 is CRITICAL: No successful Puppet run in the last 10 hours [08:14:24] PROBLEM - Puppet freshness on mw1143 is CRITICAL: No successful Puppet run in the last 10 hours [08:14:24] PROBLEM - Puppet freshness on mw1204 is CRITICAL: No successful Puppet run in the last 10 hours [08:14:25] PROBLEM - Puppet freshness on mw28 is CRITICAL: No successful Puppet run in the last 10 hours [08:15:24] PROBLEM - Puppet freshness on labstore1 is CRITICAL: No successful Puppet run in the last 10 hours [08:15:24] PROBLEM - Puppet freshness on mw1021 is CRITICAL: No successful Puppet run in the last 10 hours [08:15:24] PROBLEM - Puppet freshness on mw1104 is CRITICAL: No successful Puppet run in the last 10 hours [08:15:24] PROBLEM - Puppet freshness on mw1131 is CRITICAL: No successful Puppet run in the last 10 hours [08:15:24] PROBLEM - Puppet freshness on mw1155 is CRITICAL: No successful Puppet run in the last 10 hours [08:15:25] PROBLEM - Puppet freshness on mw1154 is CRITICAL: No successful Puppet run in the last 10 hours [08:15:38] ./manifests/role/labsmediaserver.pp [08:15:41] hahaha [08:15:41] err: Failed to apply catalog: Could not find dependency Upstart_job[twemproxy] for Service[twemproxy] at /etc/puppet/modules/mediawiki/manifests/twemproxy.pp:12 [08:15:50] speaking of the devil [08:15:53] yeah, that's what I was pointing out. separate patch [08:16:10] anyways lambsmediaserver still missing this qualified classname [08:16:10] oh, oops [08:16:24] PROBLEM - Puppet freshness on mw1047 is CRITICAL: No successful Puppet run in the last 10 hours [08:16:24] PROBLEM - Puppet freshness on mw1128 is CRITICAL: No successful Puppet run in the last 10 hours [08:16:24] PROBLEM - Puppet freshness on mw1137 is CRITICAL: No successful Puppet run in the last 10 hours [08:16:24] PROBLEM - Puppet freshness on mw1194 is CRITICAL: No successful Puppet run in the last 10 hours [08:16:24] PROBLEM - Puppet freshness on mw61 is CRITICAL: No successful Puppet run in the last 10 hours [08:16:24] or systemuser [08:16:25] PROBLEM - Puppet freshness on mw93 is CRITICAL: No successful Puppet run in the last 10 hours [08:16:25] PROBLEM - Puppet freshness on srv261 is CRITICAL: No successful Puppet run in the last 10 hours [08:16:26] PROBLEM - Puppet freshness on srv283 is CRITICAL: No successful Puppet run in the last 10 hours [08:16:26] PROBLEM - Puppet freshness on tmh1001 is CRITICAL: No successful Puppet run in the last 10 hours [08:16:26] *for [08:17:09] typing is hard... especially in the morning [08:17:24] PROBLEM - Puppet freshness on labstore2 is CRITICAL: No successful Puppet run in the last 10 hours [08:17:24] PROBLEM - Puppet freshness on mw1020 is CRITICAL: No successful Puppet run in the last 10 hours [08:17:24] PROBLEM - Puppet freshness on mw1075 is CRITICAL: No successful Puppet run in the last 10 hours [08:17:24] PROBLEM - Puppet freshness on mw1078 is CRITICAL: No successful Puppet run in the last 10 hours [08:17:24] PROBLEM - Puppet freshness on mw1179 is CRITICAL: No successful Puppet run in the last 10 hours [08:17:25] PROBLEM - Puppet freshness on mw45 is CRITICAL: No successful Puppet run in the last 10 hours [08:17:34] PROBLEM - RAID on arsenic is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:18:24] PROBLEM - Puppet freshness on mw1083 is CRITICAL: No successful Puppet run in the last 10 hours [08:18:24] PROBLEM - Puppet freshness on mw1094 is CRITICAL: No successful Puppet run in the last 10 hours [08:18:24] PROBLEM - Puppet freshness on mw1169 is CRITICAL: No successful Puppet run in the last 10 hours [08:18:24] PROBLEM - Puppet freshness on mw1136 is CRITICAL: No successful Puppet run in the last 10 hours [08:18:24] PROBLEM - Puppet freshness on mw95 is CRITICAL: No successful Puppet run in the last 10 hours [08:18:25] PROBLEM - Puppet freshness on mw99 is CRITICAL: No successful Puppet run in the last 10 hours [08:18:25] PROBLEM - Puppet freshness on srv264 is CRITICAL: No successful Puppet run in the last 10 hours [08:19:24] PROBLEM - Puppet freshness on mw1138 is CRITICAL: No successful Puppet run in the last 10 hours [08:19:24] PROBLEM - Puppet freshness on mw117 is CRITICAL: No successful Puppet run in the last 10 hours [08:19:24] PROBLEM - Puppet freshness on mw26 is CRITICAL: No successful Puppet run in the last 10 hours [08:19:24] PROBLEM - Puppet freshness on mw36 is CRITICAL: No successful Puppet run in the last 10 hours [08:20:03] ori-l: do you still need udpprofile (RT 5882) or is it going to be obsoleted by the statsd integration? [08:20:24] PROBLEM - Puppet freshness on mw1040 is CRITICAL: No successful Puppet run in the last 10 hours [08:20:24] PROBLEM - Puppet freshness on mw1062 is CRITICAL: No successful Puppet run in the last 10 hours [08:20:24] PROBLEM - Puppet freshness on mw109 is CRITICAL: No successful Puppet run in the last 10 hours [08:20:24] PROBLEM - Puppet freshness on mw107 is CRITICAL: No successful Puppet run in the last 10 hours [08:20:24] PROBLEM - Puppet freshness on mw1132 is CRITICAL: No successful Puppet run in the last 10 hours [08:20:25] PROBLEM - Puppet freshness on mw1218 is CRITICAL: No successful Puppet run in the last 10 hours [08:20:25] PROBLEM - Puppet freshness on mw123 is CRITICAL: No successful Puppet run in the last 10 hours [08:20:26] PROBLEM - Puppet freshness on mw40 is CRITICAL: No successful Puppet run in the last 10 hours [08:20:26] PROBLEM - Puppet freshness on snapshot4 is CRITICAL: No successful Puppet run in the last 10 hours [08:20:26] (03PS1) 10Ori.livneh: Upstart_job -> Generic::Upstart_job [operations/puppet] - 10https://gerrit.wikimedia.org/r/92839 [08:20:27] PROBLEM - Puppet freshness on srv296 is CRITICAL: No successful Puppet run in the last 10 hours [08:20:34] RECOVERY - RAID on arsenic is OK: OK: no RAID installed [08:21:01] paravoid: it's going to be obsoleted by the statsd integration, but it depends on how fast you want to kill professor [08:21:24] PROBLEM - Puppet freshness on mw1031 is CRITICAL: No successful Puppet run in the last 10 hours [08:21:24] PROBLEM - Puppet freshness on mw1072 is CRITICAL: No successful Puppet run in the last 10 hours [08:21:24] PROBLEM - Puppet freshness on mw1080 is CRITICAL: No successful Puppet run in the last 10 hours [08:21:24] PROBLEM - Puppet freshness on mw1134 is CRITICAL: No successful Puppet run in the last 10 hours [08:21:24] PROBLEM - Puppet freshness on mw1178 is CRITICAL: No successful Puppet run in the last 10 hours [08:21:59] it might take me 2-3 weeks [08:22:13] that's fine [08:22:24] PROBLEM - Puppet freshness on mw1059 is CRITICAL: No successful Puppet run in the last 10 hours [08:22:24] PROBLEM - Puppet freshness on mw114 is CRITICAL: No successful Puppet run in the last 10 hours [08:22:24] PROBLEM - Puppet freshness on mw15 is CRITICAL: No successful Puppet run in the last 10 hours [08:22:24] PROBLEM - Puppet freshness on mw1200 is CRITICAL: No successful Puppet run in the last 10 hours [08:22:24] PROBLEM - Puppet freshness on mw52 is CRITICAL: No successful Puppet run in the last 10 hours [08:22:25] PROBLEM - Puppet freshness on srv240 is CRITICAL: No successful Puppet run in the last 10 hours [08:22:25] PROBLEM - Puppet freshness on srv259 is CRITICAL: No successful Puppet run in the last 10 hours [08:22:26] PROBLEM - Puppet freshness on srv270 is CRITICAL: No successful Puppet run in the last 10 hours [08:22:26] PROBLEM - Puppet freshness on srv282 is CRITICAL: No successful Puppet run in the last 10 hours [08:22:28] (changeset looks good ori) [08:22:45] i should've updated the rt ticket, sorry [08:22:51] (03CR) 10Ori.livneh: [C: 032] Upstart_job -> Generic::Upstart_job [operations/puppet] - 10https://gerrit.wikimedia.org/r/92839 (owner: 10Ori.livneh) [08:22:52] thanks [08:23:24] PROBLEM - Puppet freshness on mw1 is CRITICAL: No successful Puppet run in the last 10 hours [08:23:24] PROBLEM - Puppet freshness on mw1026 is CRITICAL: No successful Puppet run in the last 10 hours [08:23:24] PROBLEM - Puppet freshness on mw1045 is CRITICAL: No successful Puppet run in the last 10 hours [08:23:24] PROBLEM - Puppet freshness on mw1082 is CRITICAL: No successful Puppet run in the last 10 hours [08:23:24] PROBLEM - Puppet freshness on mw1141 is CRITICAL: No successful Puppet run in the last 10 hours [08:23:25] PROBLEM - Puppet freshness on mw1145 is CRITICAL: No successful Puppet run in the last 10 hours [08:23:25] PROBLEM - Puppet freshness on mw1174 is CRITICAL: No successful Puppet run in the last 10 hours [08:23:26] PROBLEM - Puppet freshness on mw59 is CRITICAL: No successful Puppet run in the last 10 hours [08:23:26] PROBLEM - Puppet freshness on mw65 is CRITICAL: No successful Puppet run in the last 10 hours [08:23:27] PROBLEM - Puppet freshness on mw91 is CRITICAL: No successful Puppet run in the last 10 hours [08:23:27] PROBLEM - Puppet freshness on srv271 is CRITICAL: No successful Puppet run in the last 10 hours [08:23:37] no worries [08:24:24] PROBLEM - Puppet freshness on mw1009 is CRITICAL: No successful Puppet run in the last 10 hours [08:24:24] PROBLEM - Puppet freshness on mw1120 is CRITICAL: No successful Puppet run in the last 10 hours [08:24:24] PROBLEM - Puppet freshness on mw1060 is CRITICAL: No successful Puppet run in the last 10 hours [08:24:24] PROBLEM - Puppet freshness on mw3 is CRITICAL: No successful Puppet run in the last 10 hours [08:24:24] PROBLEM - Puppet freshness on mw7 is CRITICAL: No successful Puppet run in the last 10 hours [08:24:25] PROBLEM - Puppet freshness on srv243 is CRITICAL: No successful Puppet run in the last 10 hours [08:24:34] RECOVERY - Puppet freshness on srv271 is OK: puppet ran at Thu Oct 31 08:24:27 UTC 2013 [08:25:53] good, I see an appserver able to apply the catalog [08:26:04] RECOVERY - Puppet freshness on mw1180 is OK: puppet ran at Thu Oct 31 08:26:02 UTC 2013 [08:26:08] i tried it on srv271 and it failed it restart twemproxy [08:26:13]