[00:00:11] ah, cool, thanks [00:00:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 18.683 second response time [00:04:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [00:04:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 26.330 second response time [00:08:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [00:19:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 27.681 second response time [00:24:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [00:31:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 29.754 second response time [00:34:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [00:42:49] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 14.165 second response time [00:45:29] !log springle synchronized wmf-config/db-eqiad.php 'depool db1021 while cloning to db1045' [00:46:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [00:49:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 28.156 second response time [00:55:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [01:00:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 28.059 second response time [01:07:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [01:30:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 23.100 second response time [01:33:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: Connection timed out [01:35:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 27.471 second response time [01:46:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [01:46:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 23.942 second response time [01:54:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [01:54:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 21.464 second response time [02:00:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [02:01:46] !log LocalisationUpdate completed (1.22wmf20) at Fri Oct 11 02:01:46 UTC 2013 [02:05:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 29.699 second response time [02:08:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [02:13:55] !log LocalisationUpdate completed (1.22wmf21) at Fri Oct 11 02:13:55 UTC 2013 [02:18:46] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Oct 11 02:18:46 UTC 2013 [02:18:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 22.297 second response time [02:26:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [02:30:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 20.029 second response time [02:35:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: Connection timed out [10:49:42] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [10:50:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 25.637 second response time [10:54:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [10:55:31] (03PS1) 10TTO: Disable subpages in main namespace of eowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89177 [10:59:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 28.918 second response time [11:05:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [11:06:24] (03PS1) 10TTO: Allow bureaucrats to remove advanced permissions on wikimania2014wiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89179 [11:08:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 22.506 second response time [11:12:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [11:12:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 22.952 second response time [11:16:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [11:17:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 28.184 second response time [11:25:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [11:30:24] (03PS1) 10Dzahn: en.planet - add gsoc student feeds [operations/puppet] - 10https://gerrit.wikimedia.org/r/89184 [11:34:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 23.257 second response time [11:38:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [11:41:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 28.843 second response time [11:44:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [11:44:42] (03CR) 10Dzahn: [C: 032] en.planet - add gsoc student feeds [operations/puppet] - 10https://gerrit.wikimedia.org/r/89184 (owner: 10Dzahn) [11:52:07] (03PS4) 10Mattflaschen: Set group for /srv/mediawiki on singlenode mediawiki [operations/puppet] - 10https://gerrit.wikimedia.org/r/79955 [11:53:29] (03CR) 10Mattflaschen: "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/79955 (owner: 10Mattflaschen) [11:55:02] (03CR) 10Mattflaschen: [C: 04-1] "I tried to test this, but ran into bug 55612. I'll come back to it when that's fixed." [operations/puppet] - 10https://gerrit.wikimedia.org/r/79955 (owner: 10Mattflaschen) [11:56:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 29.927 second response time [11:56:13] (03CR) 10Mattflaschen: "I do understand this will only affect fresh clones, not existing machines on Labs." [operations/puppet] - 10https://gerrit.wikimedia.org/r/79955 (owner: 10Mattflaschen) [11:58:15] (03PS1) 10ArielGlenn: db1024 back in s7 pool warming up after upgrade/conversion to mariadb [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89194 [11:59:24] (03CR) 10ArielGlenn: [C: 032] db1024 back in s7 pool warming up after upgrade/conversion to mariadb [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89194 (owner: 10ArielGlenn) [12:00:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [12:01:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 28.318 second response time [12:02:00] !log ariel synchronized wmf-config/db-eqiad.php 'db1024 (s7) warming up after upgrade/conversion to mariadb' [12:02:18] Logged the message, Master [12:07:35] (03CR) 10Dzahn: "all the supposedly broken server are up and running and enabled in pybal" [operations/puppet] - 10https://gerrit.wikimedia.org/r/88068 (owner: 10Dzahn) [12:09:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [12:09:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 24.324 second response time [12:11:26] (03CR) 10Dzahn: [C: 032] delete dsh group "broken_appservers" [operations/puppet] - 10https://gerrit.wikimedia.org/r/88068 (owner: 10Dzahn) [12:16:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [12:47:59] PROBLEM - Puppet freshness on mw1153 is CRITICAL: No successful Puppet run in the last 10 hours [12:49:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 22.290 second response time [12:50:59] PROBLEM - Puppet freshness on mw1155 is CRITICAL: No successful Puppet run in the last 10 hours [12:53:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [12:58:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 26.444 second response time [13:01:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: Connection timed out [13:03:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 21.205 second response time [13:08:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [13:09:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 28.497 second response time [13:12:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [13:16:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 28.224 second response time [13:19:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [13:20:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 24.128 second response time [13:25:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [13:40:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 25.351 second response time [13:44:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [13:49:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 24.736 second response time [13:54:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [13:54:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 20.264 second response time [13:58:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [14:01:59] PROBLEM - Puppet freshness on cp4001 is CRITICAL: No successful Puppet run in the last 10 hours [14:05:51] (03PS1) 10Faidon Liambotis: Revert "Enable statsd reporter on [master], too." [operations/puppet] - 10https://gerrit.wikimedia.org/r/89212 [14:05:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 26.791 second response time [14:06:46] (03PS2) 10Faidon Liambotis: Revert "Enable statsd reporter on [master], too." [operations/puppet] - 10https://gerrit.wikimedia.org/r/89212 [14:07:15] (03CR) 10Faidon Liambotis: [C: 032] Revert "Enable statsd reporter on [master], too." [operations/puppet] - 10https://gerrit.wikimedia.org/r/89212 (owner: 10Faidon Liambotis) [14:07:24] (03CR) 10Faidon Liambotis: [V: 032] Revert "Enable statsd reporter on [master], too." [operations/puppet] - 10https://gerrit.wikimedia.org/r/89212 (owner: 10Faidon Liambotis) [14:09:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [14:10:49] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 14.514 second response time [14:11:28] (03PS1) 10ArielGlenn: db1024 (s7) back to normal weight in pool [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89216 [14:12:15] (03CR) 10ArielGlenn: [C: 032] db1024 (s7) back to normal weight in pool [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89216 (owner: 10ArielGlenn) [14:13:41] !log ariel synchronized wmf-config/db-eqiad.php 'db1024 (s7) back to full weight in pool' [14:13:55] Logged the message, Master [14:21:59] PROBLEM - Puppet freshness on cp4014 is CRITICAL: No successful Puppet run in the last 10 hours [14:22:59] PROBLEM - Puppet freshness on cp4019 is CRITICAL: No successful Puppet run in the last 10 hours [14:23:59] PROBLEM - Puppet freshness on cp4005 is CRITICAL: No successful Puppet run in the last 10 hours [14:23:59] PROBLEM - Puppet freshness on cp4015 is CRITICAL: No successful Puppet run in the last 10 hours [14:25:59] PROBLEM - Puppet freshness on cp4017 is CRITICAL: No successful Puppet run in the last 10 hours [14:25:59] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: No successful Puppet run in the last 10 hours [14:34:24] (03CR) 10Helder.wiki: [C: 031] (bug 54828) Enable FlaggedRevs for Portuguese Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89001 (owner: 10Odder) [14:37:59] PROBLEM - Puppet freshness on mw1064 is CRITICAL: No successful Puppet run in the last 10 hours [14:54:30] (03CR) 10Bartosz Dziewoński: Remove old Vector stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87731 (owner: 10Bartosz Dziewoński) [14:59:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [14:59:49] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 14.940 second response time [15:00:09] RECOVERY - Puppet freshness on mw1064 is OK: puppet ran at Fri Oct 11 15:00:03 UTC 2013 [15:03:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [15:14:20] Reedy: so, monday is a US holiday, moved the monday MW deploy to Tuesday morning (same time). Haven't updated /Roadmap but did wikitech:Deployments [15:14:34] Reedy: I have a headache and sick, so I'm taking the rest of the day off, just wanted to give you a heads up [15:16:09] Woo, I'm in charge! ;) [15:16:14] Hope it goes away soon [15:17:00] :) [15:17:10] paravoid: thanks for fixing that icu/php issue [15:18:06] Oh, I should probably re-rebuild collation on plwikivoyage just in case [15:18:53] done [15:18:54] xD [15:18:55] :) [15:19:17] alright, feel free to ping, I may wander by IRC a few times, [15:19:19] * greg-g waves [15:22:29] Reedy: 1.22wmf20 question for you. I was under the impression that patches backported to wmf20 on Wednesday would have hit tin/terbium yesterday, but that seems not to be the case. When will that happen? Or did I mess up and not let somebody know about something? [15:23:17] Which patches are they? [15:23:33] Specifically I was looking for my new maintenance/purgeChangedPages.php script to show up. [15:23:35] By default, I don't do any backports on deployed branches unless asked/fixing bugs [15:23:57] Let me find the gerrit for it [15:23:59] RECOVERY - Puppet freshness on mw1153 is OK: puppet ran at Fri Oct 11 15:23:52 UTC 2013 [15:24:25] Reedy: https://gerrit.wikimedia.org/r/#/c/88757/ [15:24:41] It's on tin [15:24:52] By the looks of it ori-l did the pull that checked it out [15:25:04] I'm guessing no one has explicitly sync-file'd it so it's not on terbium [15:25:34] Also your response answers my question. :) I messed up and didn't make sure you knew about it [15:25:54] You don't need me to do the deploys [15:25:58] It's on tin? I don't see it in apache/common/php-1.22wmf20/maintenance? [15:26:06] it's in /a/common [15:26:39] So it is. [15:26:54] * bd808 didn't realize that /a and /apache were different [15:26:54] !log reedy synchronized php-1.22wmf20/maintenance/purgeChangedPages.php [15:27:08] Yeah... [15:27:08] Logged the message, Master [15:27:18] Certainly in the case of maintenance scripts that aren't used in cronjobs or whatever, you can quite freely deploy those yourself, as you're not going to affect other t hings [15:28:46] Good to know. The next time I have one I'll try to get someone to watch over my shoulder so I can learn how to do that. [15:30:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 28.472 second response time [15:31:09] RECOVERY - Puppet freshness on mw1155 is OK: puppet ran at Fri Oct 11 15:31:01 UTC 2013 [15:33:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [15:35:17] RoanKattouw_away: I noticed that although the commit altering VisualEditors .gitreview file is there where I made the branch, it's not been pushed to the remote [15:36:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 28.247 second response time [15:36:12] !log removed libicu42 from apt.wikimedia.org [15:36:19] heh [15:36:23] STABSTABSTAB [15:36:24] Logged the message, Master [15:36:57] ;-/ [15:39:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [15:39:43] * MatmaRex contemplates suggesting switching en.wp to uca-en [15:42:30] apergos: 2.1GB 7zips down to 465 MB [15:43:13] still large [15:43:43] how about gz (faster) ? [15:43:56] if the difference isn't too bad it might be worth it [15:46:28] gzip is looking at 39%, 7z was around 25% [15:46:59] if it is speed you want, pigz might be better that gzip [15:47:08] and pbzip2 as well [15:47:30] is standard 7zip implementation multithreaded ? [15:47:39] Not sure [15:47:54] Sub 10 minutes on the non ADSL connection to upload that 465MB which isn't so bad [15:50:25] tried xz? although I think the algorithm is shared with at least one of the 7z algo choices [15:51:21] heh [15:51:53] Hopefully this is mostly a one off exercise.. [15:53:14] 10:2 [15:53:16] 10:20 [15:55:53] (03PS1) 10RobH: adding backup4001 info to dhcp [operations/puppet] - 10https://gerrit.wikimedia.org/r/89230 [15:59:22] (03CR) 10RobH: [C: 032] adding backup4001 info to dhcp [operations/puppet] - 10https://gerrit.wikimedia.org/r/89230 (owner: 10RobH) [16:05:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 26.390 second response time [16:09:57] The fact that the puppetlabs mysql module has a template named 'my.cnf.erb' and another named 'my.conf.cnf.erb' is almost reason enough for me to dump it right there. [16:10:50] lol [16:12:59] PROBLEM - Puppet freshness on neon is CRITICAL: No successful Puppet run in the last 10 hours [16:13:00] paravoid: i don't see anything in the ganglia graphs that suggests my change had anything to do with it actually, it only makes me think the opposite [16:13:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [16:13:21] paravoid: ie: http://ganglia.wikimedia.org/latest/graph.php?r=week&z=xlarge&h=stafford.pmtpa.wmnet&m=cpu_report&s=descending&mc=2&g=cpu_report&c=Miscellaneous+pmtpa [16:13:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 18.778 second response time [16:14:59] PROBLEM - Puppet freshness on bast4001 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:59] PROBLEM - Puppet freshness on cp4002 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:59] PROBLEM - Puppet freshness on cp4003 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:59] PROBLEM - Puppet freshness on cp4004 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:59] PROBLEM - Puppet freshness on cp4006 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:59] PROBLEM - Puppet freshness on cp4007 is CRITICAL: No successful Puppet run in the last 10 hours [16:14:59] PROBLEM - Puppet freshness on cp4008 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:00] PROBLEM - Puppet freshness on cp4009 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:00] PROBLEM - Puppet freshness on cp4010 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:01] PROBLEM - Puppet freshness on cp4011 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:01] PROBLEM - Puppet freshness on cp4012 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:02] PROBLEM - Puppet freshness on cp4013 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:02] PROBLEM - Puppet freshness on cp4016 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:03] PROBLEM - Puppet freshness on cp4018 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:03] PROBLEM - Puppet freshness on cp4020 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:04] PROBLEM - Puppet freshness on lvs4002 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:04] PROBLEM - Puppet freshness on lvs4003 is CRITICAL: No successful Puppet run in the last 10 hours [16:15:05] PROBLEM - Puppet freshness on lvs4004 is CRITICAL: No successful Puppet run in the last 10 hours [16:18:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [16:22:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 21.685 second response time [16:28:43] * Reedy wonders what rsync is doing [16:31:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [16:39:59] PROBLEM - Puppet freshness on mw1144 is CRITICAL: No successful Puppet run in the last 10 hours [16:42:21] yeah we know, gonna get to 1144 as soon as one f the other ones I'm running completes [16:42:29] * apergos grits teeth [16:44:07] (03PS1) 10ArielGlenn: run puppet at 40 minute intervals instead of 30 (try to give stafford a break) [operations/puppet] - 10https://gerrit.wikimedia.org/r/89234 [16:46:24] hmm that can't work right [16:46:27] grrrr [16:46:59] (03Abandoned) 10ArielGlenn: run puppet at 40 minute intervals instead of 30 (try to give stafford a break) [operations/puppet] - 10https://gerrit.wikimedia.org/r/89234 (owner: 10ArielGlenn) [16:47:26] an hour seems like a long time between runs too [16:49:04] fine, better an hour than multiple runs (which still turns out to be an hour or more) [16:50:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 24.234 second response time [16:54:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [16:54:20] (03PS1) 10ArielGlenn: run puppet once an hour instead of every half hour [operations/puppet] - 10https://gerrit.wikimedia.org/r/89235 [16:59:10] (03CR) 10ArielGlenn: [C: 032] run puppet once an hour instead of every half hour [operations/puppet] - 10https://gerrit.wikimedia.org/r/89235 (owner: 10ArielGlenn) [17:00:09] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 29.630 second response time [17:00:29] RECOVERY - Puppet freshness on mw1144 is OK: puppet ran at Fri Oct 11 17:00:21 UTC 2013 [17:03:06] (03PS9) 10Krinkle: wgRC2UDPPrefix: Use hostname-".org" instead of lang.site [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/47307 (owner: 10Reedy) [17:03:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [17:03:55] Reedy: https://gerrit.wikimedia.org/r/#/c/47307 ready to go? It should have no effective change, it lists all the exceptions, and changes the default. [17:07:05] Just ran into another issue with monitoring bots doing the wrong thing for testwikidata for example (testwikidata.wikipedia.org) [17:07:36] Sure [17:07:39] This needs to be fixed or the "Add a wiki" instructions updated to require an entry in wgRC2UDPPrefix. And still we should merge this so that we at least know all the exceptions, instead of them being implicit [17:08:21] All easily enough done [17:09:21] (03CR) 10Reedy: [C: 032] wgRC2UDPPrefix: Use hostname-".org" instead of lang.site [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/47307 (owner: 10Reedy) [17:09:31] (03Merged) 10jenkins-bot: wgRC2UDPPrefix: Use hostname-".org" instead of lang.site [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/47307 (owner: 10Reedy) [17:19:21] paravoid: do we still need CORS.py or can http://docs.openstack.org/developer/swift/cors.html be used to set them on containers? [17:19:45] guess that depends on the version of swift we still have [17:21:10] !log reedy synchronized wmf-config/CommonSettings.php [17:21:28] Logged the message, Master [17:21:59] !log reedy synchronized wmf-config/InitialiseSettings.php [17:22:12] Logged the message, Master [17:28:49] Reedy: got a sec to look at https://gerrit.wikimedia.org/r/89100 ? [17:28:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 24.673 second response time [17:30:14] scary url is scary ;) [17:30:23] (03PS2) 10Reedy: Whitelist Mingle Analytics RSS feed for RSS extension on Mediawiki. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89100 (owner: 10Diederik) [17:30:27] (03CR) 10Reedy: [C: 032] Whitelist Mingle Analytics RSS feed for RSS extension on Mediawiki. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89100 (owner: 10Diederik) [17:30:34] ty :) [17:32:33] (03Merged) 10jenkins-bot: Whitelist Mingle Analytics RSS feed for RSS extension on Mediawiki. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89100 (owner: 10Diederik) [17:34:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 30 seconds [17:34:49] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 17.782 second response time [17:38:52] Reedy: Thanks [17:40:20] (03PS1) 10Aaron Schulz: Add Range header support to CORS headers [operations/puppet] - 10https://gerrit.wikimedia.org/r/89238 [17:41:32] (03CR) 10jenkins-bot: [V: 04-1] Add Range header support to CORS headers [operations/puppet] - 10https://gerrit.wikimedia.org/r/89238 (owner: 10Aaron Schulz) [17:44:13] (03PS2) 10Aaron Schulz: Add Range header support to CORS headers [operations/puppet] - 10https://gerrit.wikimedia.org/r/89238 [17:44:13] whitespace nazis :) [17:45:53] hey ryan [17:46:09] any problem with rolling out ulsfo with just that localssl unified-cert only SSL gateway next week? [17:46:19] we'll change that setup later of course [17:46:29] * AaronSchulz welcomes Ryan_Lane [17:49:49] (03PS2) 10Reedy: Point php at 1.22wmf20 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89044 [17:49:54] (03CR) 10Reedy: [C: 032] Point php at 1.22wmf20 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89044 (owner: 10Reedy) [17:50:06] (03Merged) 10jenkins-bot: Point php at 1.22wmf20 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89044 (owner: 10Reedy) [17:56:42] !log Created FlaggedRevs tables on ptwiki [17:56:54] Logged the message, Master [17:56:58] (03PS2) 10Reedy: (bug 54828) Enable FlaggedRevs for Portuguese Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89001 (owner: 10Odder) [17:57:03] (03CR) 10Reedy: [C: 032] (bug 54828) Enable FlaggedRevs for Portuguese Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89001 (owner: 10Odder) [17:57:15] (03Merged) 10jenkins-bot: (bug 54828) Enable FlaggedRevs for Portuguese Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89001 (owner: 10Odder) [17:57:34] mark: no issues with that [17:57:39] good [17:57:43] mark: if you are comfortable with it [17:57:58] yeah I'll do a bit of testing on monday [17:57:59] soon we'll want to switch on a project from the beta program [17:58:16] I'd like us to make a decision on which one, announce it, then enable it soonish [17:58:41] ok [17:58:42] I think that's mostly going to give us load in esams, though [17:58:53] heh [17:59:00] i need to buy new esams boxes soon [17:59:02] !log Created EducationProgram tables on eswiki [17:59:04] not the most ideal place for that [17:59:05] so we better rollt hat in [17:59:06] yeah [17:59:14] there's still a broken ssl box there, too [17:59:17] (03PS2) 10Reedy: (bug 54826) Enable EducationProgram on the Spanish Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88639 (owner: 10Odder) [17:59:18] yeah [17:59:20] Logged the message, Master [17:59:22] (03CR) 10Reedy: [C: 032] (bug 54826) Enable EducationProgram on the Spanish Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88639 (owner: 10Odder) [17:59:35] (03Merged) 10jenkins-bot: (bug 54826) Enable EducationProgram on the Spanish Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88639 (owner: 10Odder) [18:00:34] (03PS2) 10Reedy: (bug 54223) Enable EducationProgram on the Czech Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88641 (owner: 10Odder) [18:00:38] (03CR) 10jenkins-bot: [V: 04-1] (bug 54223) Enable EducationProgram on the Czech Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88641 (owner: 10Odder) [18:02:26] (03PS3) 10Reedy: (bug 54223) Enable EducationProgram on the Czech Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88641 (owner: 10Odder) [18:02:35] (03CR) 10Reedy: [C: 032] (bug 54223) Enable EducationProgram on the Czech Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88641 (owner: 10Odder) [18:02:49] (03CR) 10jenkins-bot: [V: 04-1] (bug 54223) Enable EducationProgram on the Czech Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88641 (owner: 10Odder) [18:03:17] (03CR) 10Reedy: [V: 032] "Stupid test is stupid" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88641 (owner: 10Odder) [18:04:09] (03CR) 10Reedy: "By what branches do you mean older branches?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87731 (owner: 10Bartosz Dziewoński) [18:04:35] (03PS5) 10Reedy: User group rights configuration on Wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88694 (owner: 10Addshore) [18:04:41] (03CR) 10Reedy: [C: 032] User group rights configuration on Wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88694 (owner: 10Addshore) [18:04:54] (03Merged) 10jenkins-bot: User group rights configuration on Wikidata [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88694 (owner: 10Addshore) [18:05:04] (03PS2) 10Reedy: Disable subpages in main namespace of eowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89177 (owner: 10TTO) [18:05:10] (03CR) 10Reedy: [C: 032] Disable subpages in main namespace of eowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89177 (owner: 10TTO) [18:05:20] Reedy: by older branches i mean branches where the Vector extension is deployed and contains more than three files :P [18:05:24] (03Merged) 10jenkins-bot: Disable subpages in main namespace of eowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89177 (owner: 10TTO) [18:05:39] (or in other words, is not yet the lifeless husk i left it as) [18:05:50] (03PS2) 10Reedy: Allow bureaucrats to remove advanced permissions on wikimania2014wiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89179 (owner: 10TTO) [18:05:54] (03CR) 10Reedy: [C: 032] Allow bureaucrats to remove advanced permissions on wikimania2014wiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89179 (owner: 10TTO) [18:06:15] (03CR) 10Bartosz Dziewoński: "By older branches i mean branches where the Vector extension is deployed and contains more than three files (or in other words, is not yet" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87731 (owner: 10Bartosz Dziewoński) [18:06:34] Which is? [18:06:45] (03CR) 10Reedy: "recheck" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89179 (owner: 10TTO) [18:07:05] why am i supposed to know D: [18:07:18] let's see [18:07:40] (03Merged) 10jenkins-bot: Allow bureaucrats to remove advanced permissions on wikimania2014wiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89179 (owner: 10TTO) [18:07:55] You fixedbrokechanged it [18:08:12] Reedy: 1.20 seems to be okay, it has Vector as of 8c5d979, which is the most recent [18:09:03] wmf20* [18:09:11] (03PS3) 10Reedy: Logo configuration for *.wiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86660 (owner: 10Dereckson) [18:09:16] (03CR) 10Reedy: [C: 032] Logo configuration for *.wiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86660 (owner: 10Dereckson) [18:09:47] (03Merged) 10jenkins-bot: Logo configuration for *.wiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86660 (owner: 10Dereckson) [18:10:00] (03PS3) 10Reedy: Remove old Vector stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87731 (owner: 10Bartosz Dziewoński) [18:10:13] (03CR) 10Reedy: [C: 032] Remove old Vector stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87731 (owner: 10Bartosz Dziewoński) [18:10:21] (don't shout at me if it breaks) [18:10:27] (03Merged) 10jenkins-bot: Remove old Vector stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87731 (owner: 10Bartosz Dziewoński) [18:10:38] (03PS2) 10Reedy: (bug 48480) Remove EmailCapture extension settings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88424 (owner: 10Odder) [18:10:55] (03CR) 10Reedy: [C: 032] (bug 48480) Remove EmailCapture extension settings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88424 (owner: 10Odder) [18:12:10] (03Merged) 10jenkins-bot: (bug 48480) Remove EmailCapture extension settings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88424 (owner: 10Odder) [18:12:28] (03PS2) 10Reedy: (bug 55342) Add an alias for NS_USER_TALK on kowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88529 (owner: 10Odder) [18:12:33] (03CR) 10Reedy: [C: 032] (bug 55342) Add an alias for NS_USER_TALK on kowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88529 (owner: 10Odder) [18:12:58] (03Merged) 10jenkins-bot: (bug 55342) Add an alias for NS_USER_TALK on kowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88529 (owner: 10Odder) [18:13:26] (03PS3) 10Reedy: [CleanChanges] Set $wgCCTrailerFilter to true [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84113 (owner: 10Nemo bis) [18:13:31] (03CR) 10Reedy: [C: 032] [CleanChanges] Set $wgCCTrailerFilter to true [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84113 (owner: 10Nemo bis) [18:13:46] (03Merged) 10jenkins-bot: [CleanChanges] Set $wgCCTrailerFilter to true [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84113 (owner: 10Nemo bis) [18:14:25] (03CR) 10Reedy: [C: 04-1] "Needs rebasing" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86147 (owner: 10Yurik) [18:14:51] (03Abandoned) 10Reedy: Add Flow extension (in use on beta labs only!) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/84719 (owner: 10Spage) [18:15:26] (03PS2) 10Reedy: Clean up how VisualEditor is configured [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87644 (owner: 10Jforrester) [18:15:30] (03CR) 10Reedy: [C: 032] Clean up how VisualEditor is configured [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87644 (owner: 10Jforrester) [18:15:43] (03PS2) 10Jforrester: Switch VisualEditor to secondary status on hewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87645 [18:15:51] (03CR) 10jenkins-bot: [V: 04-1] Clean up how VisualEditor is configured [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87644 (owner: 10Jforrester) [18:15:55] (03Merged) 10jenkins-bot: Clean up how VisualEditor is configured [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87644 (owner: 10Jforrester) [18:16:21] Err. [18:16:43] V-1 / merged? [18:16:46] (03PS5) 10Reedy: Updating translation of Persian [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88108 (owner: 10Ebrahim) [18:16:50] (03CR) 10Reedy: [C: 032] Updating translation of Persian [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88108 (owner: 10Ebrahim) [18:16:52] Reedy: Is that going to be an issue? [18:17:07] (03Merged) 10jenkins-bot: Updating translation of Persian [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/88108 (owner: 10Ebrahim) [18:17:17] Reedy: https://gerrit.wikimedia.org/r/#/c/87645/ is good to go from my POV, BTW. [18:17:18] 18:15:47 PHP_Invoker_TimeoutException: Execution aborted after 1 second [18:17:24] It's jenkins failing [18:17:28] Ah, OK. [18:17:41] (03PS3) 10Reedy: Switch VisualEditor to secondary status on hewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87645 (owner: 10Jforrester) [18:17:46] (03CR) 10Reedy: [C: 032] Switch VisualEditor to secondary status on hewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87645 (owner: 10Jforrester) [18:19:26] !log reedy synchronized php-fatal-error.html [18:20:00] Logged the message, Master [18:20:01] (03CR) 10jenkins-bot: [V: 04-1] Switch VisualEditor to secondary status on hewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87645 (owner: 10Jforrester) [18:20:05] (03CR) 10Reedy: [V: 032] Switch VisualEditor to secondary status on hewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/87645 (owner: 10Jforrester) [18:20:17] !log reedy synchronized wmf-config/ [18:20:29] Logged the message, Master [18:20:50] !log reedy synchronized database lists files: [18:21:31] (03PS4) 10Reedy: Category collation for viwikivoyage to uca-vi [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85419 (owner: 10TTO) [18:21:35] (03CR) 10Reedy: [C: 032] Category collation for viwikivoyage to uca-vi [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85419 (owner: 10TTO) [18:23:01] (03CR) 10jenkins-bot: [V: 04-1] Category collation for viwikivoyage to uca-vi [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85419 (owner: 10TTO) [18:23:07] Logged the message, Master [18:23:45] (03Merged) 10jenkins-bot: Category collation for viwikivoyage to uca-vi [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85419 (owner: 10TTO) [18:24:19] mutante, thanks (for this mornings ping) [18:24:25] !log reedy synchronized wmf-config/InitialiseSettings.php [18:24:35] (03PS3) 10Reedy: (bug 54680) Set $wgCategoryCollation for the French Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86320 (owner: 10Odder) [18:24:41] (03CR) 10Reedy: [C: 032] (bug 54680) Set $wgCategoryCollation for the French Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86320 (owner: 10Odder) [18:26:10] !log Updated collation on viwikivoyage [18:26:17] !log reedy synchronized wmf-config/InitialiseSettings.php [18:26:22] Logged the message, Master [18:26:32] (03CR) 10Reedy: [V: 032] (bug 54680) Set $wgCategoryCollation for the French Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86320 (owner: 10Odder) [18:27:02] !log Updating collation on frwiki as reedy on tin in screen session. Gonna take a while [18:27:13] Logged the message, Master [18:27:13] MatmaRex: ^ I'm gonna time it out of interest [18:27:54] MatmaRex: See also https://www.mediawiki.org/wiki/User_talk:Reedy#MediaWiki_1.22.2Fwmf20.2FChangelog [18:28:48] Reedy: i'd add release notes, but extensions have no release notes [18:28:55] (let me say that there, actually) [18:29:38] I think he's wanting a breaking changes/important changes type section adding to https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf20 [18:35:24] i replied on your talk page [18:35:27] Reedy: ^ [18:36:28] <^d> Who wants a super easy puppet change? :) [18:40:45] sure [18:40:53] puppet will actually run now, if less often [18:41:05] http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=cpu_report&s=by+name&c=Miscellaneous+pmtpa&h=stafford.pmtpa.wmnet&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4 just look at these stafford graphs [18:41:26] ^d: [18:41:40] <^d> apergos: Thanks :) https://gerrit.wikimedia.org/r/#/c/84522/ [18:43:41] going to be around? I'll run it on the host (remind me which is gerrit now) so we can see if you like it [18:44:25] ^d: [18:44:28] (03PS1) 10Yuvipanda: DynamicProxy: Increase read timeout [operations/puppet] - 10https://gerrit.wikimedia.org/r/89247 [18:44:35] trivial change, can someone +2? [18:44:39] Ryan_Lane: Coren ^ [18:44:41] <^d> apergos: Yeah I'm around. It's ytterbium these days. [18:44:43] ori-l: are you subscribed to ops@ these days? [18:44:46] k sec [18:44:52] (03CR) 10ArielGlenn: [C: 032] Break lines in review Gerrit comments [operations/puppet] - 10https://gerrit.wikimedia.org/r/84522 (owner: 10Dereckson) [18:44:58] andrewbogott: i am [18:45:09] ok, I'll stop adding you as a special cc then [18:45:13] (03CR) 10jenkins-bot: [V: 04-1] DynamicProxy: Increase read timeout [operations/puppet] - 10https://gerrit.wikimedia.org/r/89247 (owner: 10Yuvipanda) [18:45:21] what now [18:45:31] (03PS2) 10Yuvipanda: DynamicProxy: Increase read timeout [operations/puppet] - 10https://gerrit.wikimedia.org/r/89247 [18:45:45] Y U NO CLICK REBASE YOURSELF, JENKINS BOT [18:46:14] andrewbogott: ^ +2? :) [18:46:32] running now [18:46:51] <^d> Cool. It's just a CSS change so the gerrit service shouldn't need restarting. [18:46:58] andrewbogott: doesn't matter either way; people sometimes do that as a manner of pinging someone [18:47:04] but thanks :) [18:47:11] YuviPanda: Ten minutes! [18:47:15] That's a hell of a long timeout [18:47:30] andrewbogott: i initially miscounted and made it 6000s :) [18:47:45] done, test please [18:47:45] andrewbogott: we apparently have tools that take more than 1m (old timeout) to return [18:47:52] andrewbogott: in this case, analytisc' wikimetrics [18:48:11] andrewbogott: I guess the actual answer is to fix the tool, but milimetric asked me to put this in for now [18:48:15] Hm, ok. Seems like this is going to cause you major future headaches when debugging things. But, I'm happy to +2 for now :) [18:48:18] ^d: [18:48:21] andrewbogott: how so? [18:48:28] andrewbogott: it's different from the *connect* timeout [18:48:37] andrewbogott: so if something doesn't connect, that'll return much faster [18:48:41] s/return/error/ [18:48:47] <^d> apergos: Everything looks good, thanks. [18:48:53] sweet [18:49:20] I guess that's it for me for the day [18:49:31] have other open small tasks but no energy left [18:49:39] 10 pm, another long day [18:49:41] (03CR) 10Andrew Bogott: [C: 032] DynamicProxy: Increase read timeout [operations/puppet] - 10https://gerrit.wikimedia.org/r/89247 (owner: 10Yuvipanda) [18:49:56] ty andrewbogott [18:52:31] * andrewbogott wonders why jenkinsbot didn't +2-verify that patch [18:52:46] baaad [18:52:47] jenkins [18:53:10] (03CR) 10Andrew Bogott: [V: 032] DynamicProxy: Increase read timeout [operations/puppet] - 10https://gerrit.wikimedia.org/r/89247 (owner: 10Yuvipanda) [18:54:58] MatmaRex: hehe, that use was one of your "allies" against some piece of JS [18:55:55] https://www.mediawiki.org/wiki/MediaWiki_1.22#Vector_extension_merged is already too big, but it would be nice to have info such as that he asks added to a page linked from there [18:57:03] Nemo_bis: {{sofixit}} plz :D [18:58:54] stupid damned vector. [18:59:26] why did anybody think it was a good idea in the first place is beyond me [19:00:11] MatmaRex: I have no idea what happened with Vector, can't write anything about it. :) [19:00:54] <^d> MatmaRex: For what it's worth, there's a number of people who complained from day 1. [19:02:38] that ain't worth much if none of these people pushed to merge them hard enough, eh [19:03:10] <^d> Meh. [19:21:49] Ryan_Lane: there better be drinks :) [19:25:58] (03PS2) 10Yurik: Removed Zero namespaces (480 & 481) from META [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86147 [19:37:32] (03CR) 10Manybubbles: [C: 031] Use new LVS setup for search [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86743 (owner: 10Chad) [19:49:38] (03PS1) 10Reedy: Remove quotewiki entries [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89326 [19:49:52] (03CR) 10Reedy: [C: 032] Remove quotewiki entries [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89326 (owner: 10Reedy) [19:51:41] (03CR) 10jenkins-bot: [V: 04-1] Remove quotewiki entries [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89326 (owner: 10Reedy) [19:52:19] (03Merged) 10jenkins-bot: Remove quotewiki entries [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89326 (owner: 10Reedy) [19:57:25] wow, that's ancient :) [19:59:33] Why have we got entries for arbcom.fi.wikimedia.org and such in the apache config? [20:01:00] :( [20:01:09] (03PS1) 10Reedy: Remove nl, fi and de arbcom.*.wikimedia.org vhosts [operations/apache-config] - 10https://gerrit.wikimedia.org/r/89329 [20:03:27] (03PS1) 10Reedy: Full compliment of misc wiki detections [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89330 [20:03:35] (03PS2) 10Reedy: Remove nl, fi and de arbcom.*.wikimedia.org vhosts [operations/apache-config] - 10https://gerrit.wikimedia.org/r/89329 [20:04:34] (03CR) 10Reedy: [C: 032] Full compliment of misc wiki detections [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89330 (owner: 10Reedy) [20:04:48] (03Merged) 10jenkins-bot: Full compliment of misc wiki detections [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89330 (owner: 10Reedy) [20:05:33] (03PS1) 10Reedy: Remove arbcom_*wiki docroots [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89331 [20:06:10] (03PS10) 10Reedy: WIP don't deduce sites based on docroot stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85165 [20:06:20] (03CR) 10jenkins-bot: [V: 04-1] WIP don't deduce sites based on docroot stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85165 (owner: 10Reedy) [20:09:39] (03PS11) 10Reedy: WIP don't deduce sites based on docroot stuff [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85165 [20:11:23] domas: https://wikitech.wikimedia.org/wiki/Ulsfo :) [20:11:47] AaronSchulz: omg, will wikipedia be faster for me?!!? [20:11:55] * Nemo_bis read that without "ls" [20:12:02] hmm, it has been that way for a year [20:12:39] https://wikitech.wikimedia.org/wiki/Ulsfo_buildout smartly forgets to say the year for deadlines :P [20:18:06] (03PS3) 10Reedy: Update tests to remove docroot setting [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85172 [20:18:17] (03CR) 10jenkins-bot: [V: 04-1] Update tests to remove docroot setting [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85172 (owner: 10Reedy) [20:20:04] (03PS4) 10Reedy: Update tests to remove docroot setting [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85172 [20:22:18] I think that code is possibly ready to be deployed.... [20:22:24] AaronSchulz: move fast and break things! [20:22:34] some of my work on el reg today! [20:22:35] http://www.theregister.co.uk/2013/10/11/flashcache_upgrade/ [20:22:36] \o/ [20:30:58] (03PS5) 10Reedy: Update tests to remove docroot setting [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85172 [20:34:54] domas: I was reading about logical read-ahead the other day [20:35:00] oh, you fb people :) [20:40:10] wow, it takes a domas to get something reading on the register [20:40:18] *something worth [20:40:59] Mituzas said. ® [20:41:20] is that a registered trademark? :D [20:43:17] domas: how does this relate to efforts like http://bcache.evilpiepirate.org/ ? [20:45:09] !log reedy synchronized wmf-config/flaggedrevs.php [20:45:19] Logged the message, Master [20:46:33] (03CR) 10Krinkle: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85172 (owner: 10Reedy) [20:50:05] gwicke: *shrug*, different ideology [20:50:12] gwicke: flashcache is easy to plug into any environment [20:50:23] no need to install some custom kernel from the future [20:50:29] maybe some day it will have lots of work done on it [20:51:29] (03PS6) 10Reedy: Update tests to remove docroot setting [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85172 [20:54:41] (03PS1) 10Reedy: Fix typo in ptwiki flagged revs config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89335 [20:55:04] (03CR) 10Reedy: [C: 032] Fix typo in ptwiki flagged revs config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89335 (owner: 10Reedy) [20:55:12] (03Merged) 10jenkins-bot: Fix typo in ptwiki flagged revs config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89335 (owner: 10Reedy) [21:10:18] Does it matter if the mysql socket file is in /run vs /tmp? [21:11:13] as long as it doesn't get removed by some tmp cleanup script [21:11:58] hm, good point. [21:19:05] domas: I'm running good old Debian which is on 3.10 and has both dm-cache and bcache [21:20:15] was looking for a solution for a hybrid drive recently, might try one of those or flashcache [21:25:00] AaronSchulz: no drinks today ;) [21:25:48] AaronSchulz: at least no special ones [21:26:00] maybe we can sit around and drink some whisky [21:28:12] Ryan_Lane: which category would a request for Cassandra test servers be? ops-requests? [21:29:05] gwicke: procurement maybe? [21:29:10] RobH: ^^ [21:29:15] ah, ok [21:29:39] so, since today is a no-deploy day... [21:29:48] should I deploy changes to the deployment system? [21:29:49] <^d> gwicke: I file everything under ops-requests and assume it'll get moved to procurement where I can't see it anymore :) [21:29:50] :D [21:29:59] <^d> Ryan_Lane: You should build my debian package ;-) [21:30:02] it seems a no-deploy day would be the best day to change deployment [21:30:14] ^d: ah, good point about not being able to see it any more [21:30:49] it also bounced me back to ops-requests when I tried to select procurement [21:31:55] heh [21:31:58] ops-requests it is [21:39:09] gwicke: procurement if you need actual bare metal servers for something [21:39:18] if you put in ops-requests though, someone will usually move it [21:40:31] haha: No permission to view newly created ticket #5948. [21:40:45] so maybe it actually ended up in procurement [21:40:47] * AaronSchulz forgot his RT password again [21:40:59] * AaronSchulz was not meant to use that by fate ;) [21:43:23] gwicke: So what kind of access are you requesting for these? [21:43:34] Also, you don't mention if you need dual cpu, or single cpu [21:43:36] ok, I'm going to merge my deployment system change [21:43:46] ori-l, gwicke: ^^ [21:43:50] also, storage space isnt really enough info [21:43:55] you two are the ones mostly using it, so I thought I'd give you a warning :) [21:43:56] how much storage space? [21:44:17] Ryan_Lane: I think you should, yeah. I can't scrutinize it in detail; I expect I'll submit patches to improve small nits I come across in the future. [21:44:25] RobH: if somebody else is happy to run the apt-get install then I'd only need normal shell access [21:44:25] * Ryan_Lane nods [21:44:33] I tested it in labs already [21:44:39] we dont do apt installs [21:44:39] so I don't expect issues [21:44:41] we do puppet installs [21:44:51] gwicke: So this has been puppetized in labs right? [21:44:53] (03CR) 10Ryan Lane: [C: 032] Change deploy repo config to repo => config [operations/puppet] - 10https://gerrit.wikimedia.org/r/88934 (owner: 10Ryan Lane) [21:45:03] RobH: no, it has not been puppetized yet [21:45:12] then isnt it a bit soon to request bare metal? [21:45:14] but I could do so [21:45:16] gwicke: some simple puppetization would be nice [21:45:20] wouldnt it be tested in labs [21:45:26] the point is to test Cassandra for storage [21:45:30] RobH: he needs to do performance testing [21:45:33] which does not work so well in labs [21:45:36] * Ryan_Lane asked him to put in a request [21:45:38] ok [21:45:50] (there are valid reasons for non labs, i dont doubt it ;) [21:46:02] my issue is we should hammer down how the machine is accessed [21:46:06] and basic setup [21:46:28] yeah, installing cassandra, node and npm with puppet should not be hard [21:46:35] we should give him access as non-root at least [21:46:41] ok [21:46:43] I'd be fine giving him root on it for now [21:46:50] i assumed thats where this was going [21:46:56] we're going to wipe them and put them back into the pool when he's done [21:46:58] cuz if he is performance tweaking, not having root is hard. [21:47:24] I just want to be able to comment on ticket during this that we discussed and such =] [21:47:24] yeah, there might be some cassandra setting tuning involved [21:47:31] gwicke: yea im not trying to stop you working mind you ;] [21:47:40] although most of those seem to be outside of the fs [21:48:01] gwicke: don't be afraid to ask for root on them ;) [21:48:10] So how much storage space you need? Also need dual cpu or single? [21:48:15] lemme link you to a page to look at [21:48:18] since they're for testing, you should be able to do what you need [21:48:20] that would certainly speed things up a bit [21:48:24] gwicke: https://wikitech.wikimedia.org/wiki/Server_Spares [21:48:29] make sure to puppetize things when necessary, though [21:48:31] thats what I have spare right now. [21:48:52] take a look and pick what is closest to spec for your needs [21:48:58] and I'll get them allocated to ya [21:49:09] (03CR) 10Ryan Lane: [C: 032] Add roles for testing sartoris in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/89126 (owner: 10Ryan Lane) [21:49:14] RobH: cool, thanks! [21:49:32] and that ticket is made correctly, so while you cannot see it (im not sure that needs to be that way) [21:49:33] RobH: is there any info on the storage of these? [21:49:47] if it doesnt mention it, its usually dual 500GB [21:49:50] only one has SAS / SSD listed [21:49:54] but some of the olders are 250GB [21:50:16] dual 500G rotating metal I guess [21:50:31] i take it back none in eqiad are 250 i dont think [21:50:39] i think its all 500s [21:50:44] on those lists. [21:50:54] gwicke: yep, with software raid [21:51:03] i have no idea if that will affect your performance testing [21:51:10] we don't need RAID really [21:51:15] !log updated git-deploy (change 88934) [21:51:17] i dont know exactly what its doing in system, so i rather give you too much info than not enough [21:51:26] could just drop one disk out of the raid though [21:51:30] we tend to raid1 [21:51:32] Logged the message, Master [21:51:37] just so a single fail doesnt offline a system [21:51:41] but as this is testing, your call. [21:51:42] cassandra can use individual disks afaik [21:51:51] would be interesting to test that [21:51:52] yea but OS has to sit someplace [21:52:09] if disk with /boot fails, cassandra wouldnt handle that would it [21:52:10] ? [21:52:13] with root I can just repurpose one of the raid disks [21:52:35] assuming both raid disks boot [21:53:11] RobH: I am curious why rt password resets do nothing, is there an error log for that? [21:53:15] RobH: the Dell PowerEdge R420 machines sound best because they also have ssds [21:53:27] AaronSchulz: i think its a known issue, the password reset thing is some addon [21:53:29] not core rt [21:53:30] * AaronSchulz should have got that in keepass...though thats win only [21:53:46] RobH: you can put xxx as a email and it goes through the same...just refreshing the page [21:53:47] gwicke: cool, I'm putting our info inthe ticket then [21:53:49] seems totally broken [21:54:02] AaronSchulz: yea, its been busted since they upgraded RT i think [21:54:12] lemme see if ther is ticket [21:54:27] heh, yep [21:54:28] https://rt.wikimedia.org/Ticket/Display.html?id=5408 [21:54:47] oh, wait, thats bz [21:55:10] oh, its right ticket, bleh [21:55:43] gwicke: ok, so those are my best servers! dont be shocked if folks ask for updates and leadtimes on how its going ;] [21:55:48] but yea, i have three you need three [21:55:51] seems ok to me. [21:56:06] RobH: the only reason is really the SSDs [21:56:16] oh, well, we can add SSDs to pretty much any of those. [21:56:22] otherwise any of the others would work too [21:56:27] cmjohnson1: has some spare SSDs on site [21:56:29] 160GB [21:56:36] that would be fine [21:57:00] well, if the 32GB memory is needed [21:57:04] rather than 16GB in lesser [21:57:07] than you can still have those [21:57:09] I don't think so [21:57:17] it would only be used for page cache [21:57:30] oh, also, cpu count [21:57:52] the 610s are dual [21:57:58] the ones im looking to hand you are single [21:58:03] is this cpu bound do you think? [21:58:13] (i realize you have to do actual bare metal testing to be sure ;) [21:58:16] as in one core only? [21:58:26] nah, just one cpu [21:58:29] lemme see how many core [21:58:51] 6 core [21:59:08] I'd have cmjohnson1 put some ssds in to three of the R320 machines [21:59:12] that might or might not be sufficient to not bottleneck on it [21:59:30] lets try it [21:59:32] well, i dont wanna fubar your testing, but seems like a good test ;] [21:59:42] if it doesn't work, we'll get some SSDs into dual cpu system for ya [21:59:43] yup, we'll find out [21:59:44] i do have ssds on site [21:59:59] cool, I'll be dropping some linked tickets for setup of three systems for gwicke [22:00:00] * cmjohnson1 scrolls back  [22:00:09] going to add SSDs to them for performance disk stuff [22:00:12] gwicke: is this the content API coming up for testing? [22:00:15] my guess is that CPU and RAM will be fine, and the SSDs and especially disks will be the bottleneck [22:00:30] YuviPanda: yes, internally for Parsoid only at first [22:00:37] right right [22:01:04] (03PS1) 10Andrew Bogott: Rearrange mysql module my.cnf defaults. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89346 [22:01:08] but if that goes well it will eventually be the backend for the content API [22:01:34] nice nice :) [22:01:38] gwicke: just one ssd per system good enough or need two? [22:01:49] we have the spares, jsut asking cuz i should. [22:01:55] 160GB [22:01:58] is ssd capacity [22:02:51] RobH: if two is easily possible then that would be great [22:02:57] yea should be fine [22:03:02] but one would also work [22:03:20] should be ok to put two, if not chris will let us know [22:03:34] RobH: awesome, thanks! [22:03:38] robh: so how many total...the problem is not the ssds but the conversion kits [22:07:15] ok, git-deploy update seemed to go well [22:07:22] time to update the docs [22:08:15] cmjohnson1: 6 ssd disks [22:08:19] we need more 3.5 to 2.5? [22:08:50] yes...i only have a few left i think 4 [22:08:55] but not more than that [22:09:02] ok, we'll have to get more [22:09:09] i'll put in procurement ticket (and link) [22:09:15] (03CR) 10Ryan Lane: [C: 032] Updates for 2.7-rc2-507-g1e7090b [operations/debs/gerrit] - 10https://gerrit.wikimedia.org/r/88129 (owner: 10Chad) [22:09:37] gwicke: so since this also includes sudo on the box, i'm dropping a linked access request ticket as well [22:09:49] you'll want to get your manager to email into it to approve later [22:09:58] just so we follow all the procedure, but its formality. [22:39:40] RobH: ok, thanks [22:40:00] that manager just increased my latency a bit [23:16:24] no worries, i dont imagine the ssds will be in before tuesday [23:16:33] so plenty of time for associated tickets to get resolved [23:16:39] (03PS1) 10Ori.livneh: Add varnish::logging::client_stats [operations/puppet] - 10https://gerrit.wikimedia.org/r/89359 [23:43:32] (03PS1) 10Yuvipanda: Gerrit: More useful subject line for Gerrit changesets [operations/puppet] - 10https://gerrit.wikimedia.org/r/89365 [23:43:34] ^d: ^ [23:44:27] <^d> Heh [23:44:44] (03CR) 10Yuvipanda: [C: 04-1] "Could be more informative" [operations/puppet] - 10https://gerrit.wikimedia.org/r/89365 (owner: 10Yuvipanda) [23:44:47] YuviPanda: "The needful"? :-) [23:45:08] YuviPanda: No English variants in gerrit mail, please. :-) [23:45:14] James_F: https://en.wikipedia.org/wiki/Do_the_needful [23:45:25] James_F: it's on enwiki! It obviously must be acceptable. [23:45:27] YuviPanda: I'm aware of the idiom. [23:45:35] YuviPanda: :-P [23:46:29] James_F: :D who wouldn't want thousands of emails from gerrit with the same subject line... [23:46:37] (03CR) 10jenkins-bot: [V: 04-1] Gerrit: More useful subject line for Gerrit changesets [operations/puppet] - 10https://gerrit.wikimedia.org/r/89365 (owner: 10Yuvipanda) [23:46:38] YuviPanda: :-D [23:46:45] bad, BAD JENKINS BOT [23:49:08] (03PS2) 10Yuvipanda: Gerrit: More useful subject line for Gerrit changesets [operations/puppet] - 10https://gerrit.wikimedia.org/r/89365 [23:49:10] James_F: you might find the new patchset more amenable to your tastes. [23:49:52] YuviPanda: :-P [23:50:24] (03Abandoned) 10Yuvipanda: Gerrit: More useful subject line for Gerrit changesets [operations/puppet] - 10https://gerrit.wikimedia.org/r/89365 (owner: 10Yuvipanda) [23:51:31] <^d> YuviPanda: You need to combine yours and James_F more. Something like "Please have a jolly good time doing the needful." [23:51:40] ^d: … What What?! [23:51:44] hahah [23:51:51] I Say! Good Show! [23:52:19] 'bloody hell, jenkins, what now!?!' [23:52:31] i don't know if that's british, actually [23:52:33] or australian [23:53:15] <^d> I dunno. Ya'll be talkin' too high-falootin' for the likes a'me. [23:53:39] <^d> That was painful to type. [23:53:43] hehe [23:54:00] my british english knowledge is pretty bad right now because a lot of it is from Jon Robson who punctuates every sentence with a laugh