[00:49:32] (03PS2) 10Krinkle: robots.php: Use max() time and clean up [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177996 [00:50:35] (03PS3) 10Krinkle: robots.php: Simplify Last-Modified logic and re-use code [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177996 [00:50:39] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 11 data above and 0 below the confidence bounds [00:51:09] (03PS4) 10Krinkle: robots.php: Simplify Last-Modified logic and re-use code [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177996 [00:53:18] (03PS5) 10Krinkle: robots.php: Simplify Last-Modified logic and re-use code [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177996 [00:54:32] (03CR) 10Krinkle: [C: 032] robots.php: Simplify Last-Modified logic and re-use code [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177996 (owner: 10Krinkle) [00:54:57] (03Merged) 10jenkins-bot: robots.php: Simplify Last-Modified logic and re-use code [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177996 (owner: 10Krinkle) [00:58:51] !log krinkle Synchronized w/robots.php: 611892c62349d09c9758 (duration: 00m 06s) [00:58:59] Logged the message, Master [01:20:25] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: Puppet has 1 failures [01:46:00] Krinkle, have oyu tested that robots.php thingie? ;) [01:46:59] MaxSem: Yes. [01:47:12] Uh? [01:47:15] Not anymore apparently [01:47:17] That's weird [01:47:24] Krinkle, now look in logstash;) [01:47:40] that's why we don't deploy on fridays:P [01:47:48] Don't have to, got closer-to-the-metal alerts already [01:47:50] reverting/fixing [01:51:17] (03CR) 10Legoktm: Don't collapse sections on mobile WD (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179513 (owner: 10MaxSem) [01:53:32] (03PS1) 10Krinkle: robots.php: Fixup E_NOTICE undefined $stats [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179587 [01:53:53] (03PS2) 10Krinkle: robots.php: Fix 'Notice: Undefined stats' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179587 [01:54:53] (03CR) 10Krinkle: [C: 032] robots.php: Fix 'Notice: Undefined stats' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179587 (owner: 10Krinkle) [01:55:02] (03Merged) 10jenkins-bot: robots.php: Fix 'Notice: Undefined stats' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179587 (owner: 10Krinkle) [01:56:23] !log krinkle Synchronized w/robots.php: 54746fdef3402 (duration: 00m 05s) [02:09:24] !log l10nupdate Synchronized php-1.25wmf11/cache/l10n: (no message) (duration: 00m 01s) [02:09:28] !log LocalisationUpdate completed (1.25wmf11) at 2014-12-13 02:09:28+00:00 [02:09:32] Logged the message, Master [02:09:40] Logged the message, Master [02:13:56] !log l10nupdate Synchronized php-1.25wmf12/cache/l10n: (no message) (duration: 00m 01s) [02:14:00] !log LocalisationUpdate completed (1.25wmf12) at 2014-12-13 02:14:00+00:00 [02:14:06] Logged the message, Master [02:14:12] Logged the message, Master [02:14:43] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:25:00] RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected [02:48:39] PROBLEM - Host amssq52 is DOWN: CRITICAL - Plugin timed out after 15 seconds [02:48:45] PROBLEM - Host amssq34 is DOWN: CRITICAL - Plugin timed out after 15 seconds [02:48:45] PROBLEM - Host amssq45 is DOWN: CRITICAL - Plugin timed out after 15 seconds [02:48:45] PROBLEM - Host amssq35 is DOWN: CRITICAL - Plugin timed out after 15 seconds [02:48:58] RECOVERY - Host amssq52 is UP: PING OK - Packet loss = 0%, RTA = 96.31 ms [02:49:04] RECOVERY - Host amssq45 is UP: PING OK - Packet loss = 0%, RTA = 95.54 ms [02:49:11] RECOVERY - Host amssq35 is UP: PING OK - Packet loss = 0%, RTA = 95.39 ms [02:49:13] RECOVERY - Host amssq34 is UP: PING OK - Packet loss = 0%, RTA = 95.47 ms [03:29:31] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Dec 13 03:29:31 UTC 2014 (duration 29m 30s) [03:29:40] Logged the message, Master [03:37:27] PROBLEM - puppet last run on mw1238 is CRITICAL: CRITICAL: Puppet has 1 failures [03:50:06] RECOVERY - puppet last run on mw1238 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [04:48:06] (03PS1) 10Andrew Bogott: Whitespace change in virt-hp.cfg partman recipe. [puppet] - 10https://gerrit.wikimedia.org/r/179589 [04:49:34] (03CR) 10Andrew Bogott: [C: 032] Whitespace change in virt-hp.cfg partman recipe. [puppet] - 10https://gerrit.wikimedia.org/r/179589 (owner: 10Andrew Bogott) [05:52:38] (03PS2) 10KartikMistry: Added initial Debian packaging [debs/contenttranslation/hfst] - 10https://gerrit.wikimedia.org/r/179153 [05:59:00] PROBLEM - puppet last run on mw1185 is CRITICAL: CRITICAL: Puppet has 1 failures [06:08:13] PROBLEM - puppet last run on mw1252 is CRITICAL: CRITICAL: Puppet has 1 failures [06:14:14] RECOVERY - puppet last run on mw1185 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:23:12] RECOVERY - puppet last run on mw1252 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:33:49] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: puppet fail [06:34:28] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:41] PROBLEM - puppet last run on db1003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:45] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:56] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:09] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:38:30] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:46:26] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:46:27] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:46:41] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:03] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:30] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:49:02] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:49:43] RECOVERY - puppet last run on db1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:03:39] (03CR) 1020after4: [C: 031] "I also think ensure=>latest makes most sense for CI." [puppet] - 10https://gerrit.wikimedia.org/r/178806 (owner: 10Hashar) [07:15:46] PROBLEM - puppet last run on mw1151 is CRITICAL: CRITICAL: Puppet has 1 failures [07:30:37] RECOVERY - puppet last run on mw1151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:34:53] (03CR) 10Aaron Schulz: "Actually imagescalers are still in progress" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/178591 (owner: 10Aaron Schulz) [08:24:59] (03CR) 10Faidon Liambotis: [C: 04-1] "It might, but let's not make ensure conditional to the realm across all of our puppet code. If CI wants to have latest & greatest, it shou" [puppet] - 10https://gerrit.wikimedia.org/r/178806 (owner: 10Hashar) [08:28:11] (03CR) 10Faidon Liambotis: [C: 031] "Right, that's obviously correct (although I read that 3.5's future parser/4.0 will support this syntax)." [puppet] - 10https://gerrit.wikimedia.org/r/179472 (owner: 10Giuseppe Lavagetto) [08:31:25] (03PS1) 10Yuvipanda: salt: Use fqdn as client id for labs as well [puppet] - 10https://gerrit.wikimedia.org/r/179592 [08:31:54] (03CR) 10Yuvipanda: [C: 04-2] "Block until we figure out how to safely do this." [puppet] - 10https://gerrit.wikimedia.org/r/179592 (owner: 10Yuvipanda) [08:36:33] (03CR) 10Nikerabbit: [C: 031] Set wgTranslateTranslationServices['TTMServer']['cutoff'] [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179566 (owner: 10BryanDavis) [09:30:50] PROBLEM - puppet last run on mw1106 is CRITICAL: CRITICAL: Puppet has 1 failures [09:46:01] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [09:51:50] !log Restarting Jenkins to get rid of some deadlocks that occurred yesterday [09:51:55] Logged the message, Master [10:05:02] PROBLEM - DPKG on lanthanum is CRITICAL: DPKG CRITICAL dpkg reports broken packages [10:37:16] (03PS1) 10Dereckson: Enabled NewUserMessage on fa.wikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179596 [11:51:20] PROBLEM - puppet last run on mw1242 is CRITICAL: CRITICAL: Puppet has 1 failures [11:59:25] PROBLEM - puppetmaster https on virt1000 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [12:06:35] RECOVERY - puppet last run on mw1242 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [12:15:32] (03PS1) 10Andrew Bogott: Another long-shot partman attempt. [puppet] - 10https://gerrit.wikimedia.org/r/179600 [12:17:21] (03CR) 10Andrew Bogott: [C: 032] Another long-shot partman attempt. [puppet] - 10https://gerrit.wikimedia.org/r/179600 (owner: 10Andrew Bogott) [12:20:49] RECOVERY - puppetmaster https on virt1000 is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.025 second response time [12:20:49] !log graceful'd apache2 on virt1000; puppet master was acting up. [12:20:52] Logged the message, Master [12:45:34] PROBLEM - puppet last run on amssq52 is CRITICAL: CRITICAL: puppet fail [12:48:12] andrewbogott_afk: yeah that won't work.. [12:48:23] xvdb for starters [12:51:10] andrewbogott_afk: you can partition/format sdb after boot though [12:51:13] manually or with puppet [12:51:24] that's what we do with swift, there's no reason for this to happen in the installer [12:51:41] the installer should do the minimal thing needed to have a functional booting system [13:00:42] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:31:14] PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: Puppet has 1 failures [13:44:05] PROBLEM - puppet last run on mw1220 is CRITICAL: CRITICAL: Puppet has 1 failures [13:46:25] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:56:08] RECOVERY - puppet last run on mw1220 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [14:10:01] PROBLEM - puppet last run on mw1240 is CRITICAL: CRITICAL: Puppet has 1 failures [14:24:57] RECOVERY - puppet last run on mw1240 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:35:37] <_joe_> I've had enough of those puppet failures. Let's fix this. [14:35:47] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: Puppet has 1 failures [14:36:18] PROBLEM - puppet last run on mw1258 is CRITICAL: CRITICAL: Puppet has 1 failures [14:41:12] (03PS1) 10Giuseppe Lavagetto: hhvm: remove jemalloc profiling completely [puppet] - 10https://gerrit.wikimedia.org/r/179609 [14:48:00] RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:51:17] RECOVERY - puppet last run on mw1258 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:52:07] (03PS1) 10Giuseppe Lavagetto: hhvm: fix tidy perf maps cronjob [puppet] - 10https://gerrit.wikimedia.org/r/179612 [14:53:44] (03CR) 10Giuseppe Lavagetto: [C: 032] hhvm: fix tidy perf maps cronjob [puppet] - 10https://gerrit.wikimedia.org/r/179612 (owner: 10Giuseppe Lavagetto) [15:01:29] (03PS1) 10Giuseppe Lavagetto: hhvm: run the tidy cron once per day... [puppet] - 10https://gerrit.wikimedia.org/r/179613 [15:02:50] PROBLEM - puppet last run on mw1091 is CRITICAL: CRITICAL: Puppet has 1 failures [15:02:54] <_joe_> how many WTFs can we create per puppet line of code? [15:03:43] (03CR) 10Giuseppe Lavagetto: [C: 032] hhvm: run the tidy cron once per day... [puppet] - 10https://gerrit.wikimedia.org/r/179613 (owner: 10Giuseppe Lavagetto) [15:09:39] (03CR) 10Giuseppe Lavagetto: "when I misspelled a node name in the puppet compiler, so that $::lsbdistid was empty :)" [puppet] - 10https://gerrit.wikimedia.org/r/179472 (owner: 10Giuseppe Lavagetto) [15:14:56] RECOVERY - puppet last run on mw1091 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [15:43:48] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [16:09:42] PROBLEM - puppet last run on mw1244 is CRITICAL: CRITICAL: Puppet has 1 failures [16:11:46] <_joe_> paravoid: jessie has puppet 3.7, cool [16:21:49] RECOVERY - puppet last run on mw1244 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:59:59] (03PS2) 10BryanDavis: Ensure that apache's uid=48 [puppet] - 10https://gerrit.wikimedia.org/r/178690 [17:00:17] (03CR) 10BryanDavis: "Cherry-picking to beta for testing" [puppet] - 10https://gerrit.wikimedia.org/r/178690 (owner: 10BryanDavis) [18:37:49] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: Puppet has 1 failures [18:52:51] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:56:38] _joe_: the puppet failures were jemalloc-related? [18:57:07] oh, the curl. [18:57:34] yeah, i'll e-mail the jemalloc list about it. i think you did the right thing. [19:15:57] PROBLEM - puppet last run on mw1149 is CRITICAL: CRITICAL: Puppet has 1 failures [19:30:58] RECOVERY - puppet last run on mw1149 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:57:22] PROBLEM - puppet last run on mw1163 is CRITICAL: CRITICAL: Puppet has 1 failures [20:05:06] PROBLEM - puppet last run on lvs2002 is CRITICAL: CRITICAL: puppet fail [20:12:18] RECOVERY - puppet last run on mw1163 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:19:51] RECOVERY - puppet last run on lvs2002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:46:00] PROBLEM - puppet last run on mw1035 is CRITICAL: CRITICAL: Puppet has 1 failures [22:01:12] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:15:51] PROBLEM - puppet last run on mw1051 is CRITICAL: CRITICAL: Puppet has 1 failures [22:21:57] (03CR) 10Legoktm: [C: 04-1] "Pretty sure this won't work." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/179094 (owner: 10Steinsplitter) [22:30:52] RECOVERY - puppet last run on mw1051 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:17:59] (03PS1) 10BryanDavis: logstash: port udp2log rules to monolog input [puppet] - 10https://gerrit.wikimedia.org/r/179758 [23:18:24] <_joe_> ori: I didn't merge the change btw [23:18:38] _joe_: i lost context [23:19:04] oh, the jemalloc thing [23:19:10] _joe_: feel free to merge it [23:23:50] (03CR) 10BryanDavis: "Cherry-picked to deployment-salt for testing" [puppet] - 10https://gerrit.wikimedia.org/r/179758 (owner: 10BryanDavis) [23:33:25] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: Puppet has 1 failures [23:35:50] <_joe_> ori: http://wpengine.com/2014/11/19/hhvm-project-mercury/ apparently our "retry on zend" varnish patch was 'enterprise high availability :P [23:37:19] _joe_: you joke, but if we weren't a pair of stupid socialists with bleeding heart for the cause, we'd start an hhvm migration consultancy and make money like bandits [23:37:45] <_joe_> eheh [23:37:51] That would be easy wouldn't it. [23:38:01] * bd808 makes some calls [23:38:05] heh. [23:38:48] * ori weekend, bbl [23:39:58] (at times like this, i'm sad that there's no place to submit quips to) [23:41:40] let’s make on! [23:41:42] MatmaRex: Sounds like a job for toollabs [23:41:43] *one [23:42:01] bash.wmflabs.org [23:42:10] tools.wmflabs.org/bash [23:42:45] * YuviPanda starts new RoR project [23:43:02] y’know, making HHVM projects easy to use for toollabs probably isn’t such a bad idea [23:43:06] * bd808 throws up in his mouth a little [23:43:15] RoR project [23:43:19] most tools are badly written enough that they restart once every few hours anyway [23:44:18] but I’m pretty sure valhallasw will kill me if I do that before doing uwsgi support [23:44:20] * YuviPanda files away [23:45:06] I should find a tool to help out with one of these days just to learn how the whole system works [23:48:41] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:49:16] (03CR) 10Andrew Bogott: "This looks ok to me, although I'm not clear on what will happen when this applies on a system with an existing non-48 Apache user. Have y" [puppet] - 10https://gerrit.wikimedia.org/r/178690 (owner: 10BryanDavis) [23:52:34] (03CR) 10BryanDavis: "I have not, but it should renumber the user. In beta I have audited and found that this shouldn't happen (all users are uid=48 now). In pr" [puppet] - 10https://gerrit.wikimedia.org/r/178690 (owner: 10BryanDavis) [23:58:02] bd808: +1, you can even build the quips one :D *hint hint* *nudge nudge* [23:59:42] <_joe_> directly from /r/lolphp, http://news.php.net/php.internals/79446