[00:00:29] PROBLEM - Puppet freshness on cp4013 is CRITICAL: Last successful Puppet run was Mon 02 Dec 2013 11:58:44 PM UTC [00:00:33] mutante: cool., thanks! [00:00:46] (03CR) 10Ori.livneh: [C: 032] Add logstash100[1-3] to site.pp & add bd808 & aaron as sudo per RT 6366 [operations/puppet] - 10https://gerrit.wikimedia.org/r/98730 (owner: 10Ori.livneh) [00:01:15] wee [00:01:54] gonna manually run puppet on those nodes [00:02:04] btw, who was that se4598 person? [00:02:06] mwalker: ? [00:02:29] PROBLEM - Puppet freshness on cp4013 is CRITICAL: Last successful Puppet run was Mon 02 Dec 2013 11:58:44 PM UTC [00:02:44] possibly steinsplitter from https://gerrit.wikimedia.org/r/#/c/98073/ [00:03:11] greg-g: http://en.wikipedia.org/wiki/User:Se4598 [00:03:12] ahh, yeah, figured it out now, thanks mwalker [00:03:30] I'm going to push my config change now [00:04:05] mwalker: cool [00:04:20] !log mwalker updated /a/common to {{Gerrit|I9e2f24923}}: Changing banner expiration to 10 months [00:04:29] oh [00:04:29] PROBLEM - Puppet freshness on cp4013 is CRITICAL: Last successful Puppet run was Mon 02 Dec 2013 11:58:44 PM UTC [00:04:30] that's new [00:04:35] Logged the message, Master [00:05:13] !log mwalker synchronized wmf-config/InitialiseSettings.php [00:05:16] mwalker: yeah, meant to send an email about that [00:05:28] Logged the message, Master [00:05:35] :) [00:05:42] its kinda cool [00:05:45] !log mwalker synchronized wmf-config/CommonSettings.php [00:05:46] mwalker: btw, i just ran across rules in AdBlock against Wikipedia banners [00:05:59] Logged the message, Master [00:06:07] mutante: if it's just the extended rules; we're aware [00:07:14] mwalker: https://adblockplus.org/forum/viewtopic.php?f=2&t=19557&p=87593&hilit=wikipedia#p87593 [00:07:36] |http://meta.wikimedia.org/wiki/Special:BannerRandom?* [00:07:52] it's a request they didnt reply to yet [00:10:39] PROBLEM - Puppet freshness on sq37 is CRITICAL: Last successful Puppet run was Sun 01 Dec 2013 06:02:06 AM UTC [00:11:20] well; if they do; we're already in the EasyPrivacy list... so... [00:11:27] it's not good [00:11:33] but I don't know what else to do [00:16:48] mwalker: i wouldn't suggest you try antiblock.org , that just makes people use Disable Anti-Adblock .. and anti-anti-anti-anti :p [00:17:26] heh; no [00:17:38] basically; if people want to block us; that's their perogative [00:17:51] and in a month I'm going to make it significantly harder to do so on all wikis [00:17:55] by changing the URLs [00:18:11] but... even then [00:18:14] it is technically an ad [00:18:17] mwalker: more seriously, what you can do is try get on the "accepted ads" list on ABP [00:18:28] but then, i know how controversial that feature is [00:18:33] and AdBlockPlus itself [00:19:02] mwalker: yea, the only way is to follow their guidelines to be accepted as non-intrusive ad [00:19:31] and _supposedly_ people paid for that .. (and AdBlockPlus != AdBlock or AdBlockEdge, which dont have that feature) [00:21:18] https://adblockplus.org/forum/viewforum.php?f=17 [00:21:58] http://searchenginewatch.com/article/2280451/Google-Paying-to-Have-Ads-Whitelisted-on-AdBlock-Plus [00:23:16] (03PS1) 10Ori.livneh: Add accounts for bd808 and aaron on logstash nodes [operations/puppet] - 10https://gerrit.wikimedia.org/r/98736 [00:26:11] (03CR) 10Ori.livneh: [C: 032] Add accounts for bd808 and aaron on logstash nodes [operations/puppet] - 10https://gerrit.wikimedia.org/r/98736 (owner: 10Ori.livneh) [00:28:59] RECOVERY - Puppet freshness on cp4013 is OK: puppet ran at Tue Dec 3 00:28:57 UTC 2013 [00:29:49] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [00:40:14] ^d: I'm going to go to the store. when I get back I can take over script babysitting [00:40:33] <^d> Mmk. [00:40:35] I believe I saw an error building one of the wikis. testwikidata or something. I'll figure it out when I get back [00:45:07] greg-g: dumbish question; I don't need to do any mystic voodoo beyond a sync-file to get a configuration change applied? [00:47:54] (03PS1) 10BryanDavis: [WIP] Add configuration for Wikimania Scholarships [operations/puppet] - 10https://gerrit.wikimedia.org/r/98740 [00:48:49] (03CR) 10jenkins-bot: [V: 04-1] [WIP] Add configuration for Wikimania Scholarships [operations/puppet] - 10https://gerrit.wikimedia.org/r/98740 (owner: 10BryanDavis) [00:51:03] (03PS2) 10BryanDavis: [WIP] Add configuration for Wikimania Scholarships [operations/puppet] - 10https://gerrit.wikimedia.org/r/98740 [00:56:21] mwalker: aside from the rain dance you mean? [00:56:26] mwalker: that should be it [00:56:35] heh -- ya; it turns out I was looking at cached content [01:08:24] (03CR) 10BryanDavis: [WIP] Add configuration for Wikimania Scholarships (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/98740 (owner: 10BryanDavis) [01:13:17] !log rebooting sq37 [01:13:34] Logged the message, Master [01:16:00] PROBLEM - Host sq37 is DOWN: PING CRITICAL - Packet loss = 100% [01:23:36] cmjohnson1: fyi.. https://gerrit.wikimedia.org/r/#/c/94115/ & https://gerrit.wikimedia.org/r/#/c/96489/ that's why i didnt make a new one [01:24:01] not saying it needs merge now. but re: tesla [01:24:55] mutante: what do you think about the comment? [01:25:08] regarding the virtualization subnet? [01:26:51] cmjohnson1: pretty much exactly what ariel commented :) [01:27:11] (03CR) 10Ori.livneh: [C: 04-1] "Comments inline. I think the Apache config deserves a lookover from someone more familiar with setting up PHP webapps." (0312 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/98740 (owner: 10BryanDavis) [01:27:12] could make a new patchset that just removes actual tesla [01:38:52] http://git.wikimedia.org/ is (almost) dead, long live git [01:47:01] (03PS3) 10BryanDavis: [WIP] Add configuration for Wikimania Scholarships [operations/puppet] - 10https://gerrit.wikimedia.org/r/98740 [01:49:34] TimStarling: i want to depool es1004. it is as simple as commenting out the IP in $wmgOldExtTemplate ? safe, etc [01:51:47] <^d> manybubbles|away: Almost done for all wikis but cawiki, enwikisource, frwikisource, nlwiki, itwiki (these 5 are still on pass 1) [01:53:11] (03CR) 10BryanDavis: [WIP] Add configuration for Wikimania Scholarships (0311 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/98740 (owner: 10BryanDavis) [01:59:42] (03CR) 10Dzahn: "2 minor puppet-lint warnings inline and please retab to use 4 spaces instead of actual tabs. i know we have it both ways, but tabs are the" (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/98740 (owner: 10BryanDavis) [02:02:06] (03PS4) 10Dzahn: [WIP] Add configuration for Wikimania Scholarships [operations/puppet] - 10https://gerrit.wikimedia.org/r/98740 (owner: 10BryanDavis) [02:03:03] ^d cool [02:03:07] thanks. [02:03:11] im back [02:03:16] <^d> no problem :) [02:03:27] everything good? [02:03:30] <^d> Yep. [02:03:33] <^d> Humming along dandy. [02:04:01] kids have hand foot and mouth so i went to get groeries and drugs [02:05:17] ori-l, you will be happy to know that i began working on JsonConfig ext... will see what i will come up with :) [02:06:36] <^d> manybubbles|away: Take care of family. I've got this. It's completely uneventful. [02:08:01] yurik-road: cool, looking forward [02:08:11] yurik-road: i still owe you a review of the vagrant patch [02:08:18] ori-l, want to creat a repo for me btw? [02:08:24] i submitted req [02:08:32] its on the repo request page somewhere [02:08:40] mediawiki/extensions/JsonConfig? [02:08:50] yep [02:09:15] do you know whom i should talk to about db schema deployment? This is not new schema - rather i'm trying to deploy an existing extension (flag revs) to a new site (metawiki) [02:09:41] do we have a special procedure for that? [02:10:39] yurik-road: springle i suppose [02:11:15] (03PS7) 10Dzahn: bugzilla module [operations/puppet] - 10https://gerrit.wikimedia.org/r/94075 [02:12:44] PROBLEM - Puppet freshness on cp3012 is CRITICAL: Last successful Puppet run was Mon 02 Dec 2013 05:10:45 PM UTC [02:14:08] mutante, thx, will try to get to springle [02:16:44] PROBLEM - Puppet freshness on cp3011 is CRITICAL: Last successful Puppet run was Mon 02 Dec 2013 05:15:19 PM UTC [02:17:26] (03CR) 10Dzahn: [C: 031] "Alex, even more comments now on PS4, i think i fixed pretty much all, now also unified those 2 scripts into a single define and it runs on" [operations/puppet] - 10https://gerrit.wikimedia.org/r/94075 (owner: 10Dzahn) [02:17:50] !log LocalisationUpdate completed (1.23wmf4) at Tue Dec 3 02:17:50 UTC 2013 [02:17:57] yurik-road: https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/JsonConfig [02:18:00] ^d: k. send me an email about where things are when you are done for the night. I imagine some stuff will run over night now that we're sure it is uneventful. [02:18:04] can you mark the request as fulfilled? I gotta run [02:18:08] Logged the message, Master [02:18:14] <^d> manybubbles|away: Will do, have a good night [02:18:21] you too [02:18:28] ori-l, thx!!! [02:18:44] akosiaris: ^ when you see this later, replied to all the review comments re: BZ, see above [02:18:56] gerrit 94075 [02:27:01] yurik-road: whats up? [02:27:20] schema stuff? [02:27:27] hi springle, i need to deploy flagged revs extension on meta [02:27:36] wasn't sure whom to bug about db update [02:27:57] new tables or alterations, or both? [02:28:22] no idea - the extension is out there on many sites [02:28:22] like de, etc [02:28:29] looking... [02:29:24] https://wikitech.wikimedia.org/wiki/FlaggedRevs_setup has some old docs from 2011 [02:29:33] springle, the config change patch has a link https://gerrit.wikimedia.org/r/#/c/95662/ [02:29:42] http://git.wikimedia.org/blob/operations%2Fmediawiki-config.git/a8723d447344c57a4f40b52eae076c683201a11a/wmf-config%2FInitialiseSettings.php#L10138 [02:30:21] yurik, Reedy is probably a good person to help if you can wait til he's around [02:31:54] Eloquence, no rush, just need to figure out whom to bug at the end of the day :) [02:32:10] (and not the literal end of the day) [02:33:33] yurik-road: generally devs (or yeah, reedy) can deploy new tables for extensions themselves. i want to know about it if schema alterations are happening so they can be done online or have downtime scheduled [02:34:11] well, that's my point - i didn't want to deploy it without knowing the process first :) [02:34:19] if data updates occur, the jobs need to be batched, but also ok for reedy-eqsue devs to run :) [02:34:43] ppl might get unhappy if meta goes down [02:35:11] heh, then yeah, ask reedy or someone who's deployed flaggedrevs before. not i, so far (i'd wait for reedy too) [02:35:49] or aaron [02:37:13] <^d> Yes, those instructions are pretty much accurate. [02:37:20] <^d> Obviously s/1.18/something modern/ [02:37:37] !log LocalisationUpdate completed (1.23wmf5) at Tue Dec 3 02:37:37 UTC 2013 [02:37:53] Logged the message, Master [02:37:56] thx springle [02:41:31] (03PS1) 10Springle: depool es1004 for upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98749 [02:42:23] (03CR) 10Springle: [C: 032] depool es1004 for upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98749 (owner: 10Springle) [02:43:26] !log springle synchronized wmf-config/db-eqiad.php 'depool es1004 for upgrade' [02:43:40] Logged the message, Master [03:07:06] PROBLEM - check_job_queue on terbium is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:07:56] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [03:19:58] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Dec 3 03:19:58 UTC 2013 [03:20:06] TimStarling: i want to depool es1004. it is as simple as commenting out the IP in $wmgOldExtTemplate ? safe, etc [03:20:13] Logged the message, Master [03:20:25] obviously it's a bit late now since you just did it, but it's reasonably safe, yes [03:21:17] job queue runners will keep going with the same configuration for a few minutes or so after the configuration change [03:22:45] which matters a little bit more with external storage than with core DBs, since external storage is slightly more likely to give soft (non-exception) failures [03:23:33] TimStarling, hi, do you know by any chance why i don't see zero extension in https://graphite.wikimedia.org/dashboard/ ? [03:24:03] or is there a better place to look at profiling data [03:24:36] what profiling sections are there? [03:25:13] TimStarling, there are hundreds [03:25:40] I see some at http://noc.wikimedia.org/cgi-bin/report.py?db=all&sort=real&limit=5000 [03:25:46] maybe the backslashes confuse graphite [03:26:00] oh, TimStarling veryy sorry, i found it under ExtZero... [03:26:13] not sure why it was defined like that [03:26:30] and thx for the link [03:38:47] TimStarling: thanks. i spent a while digging through code to come to that conclusion. es1004 is depooled and waiting for wikiadmin connnections to die [03:39:05] taking longer than a few mins though. sleepers [03:39:51] are you taking the server down, or doing some other kind of maintenance on it? [03:40:10] package upgrade including mariadb I hope [03:41:01] hmm, ExternalStoreDB.php needs some updates for PHP 5 [03:41:15] I guess this code doesn't get looked at very often [03:41:16] er, LTS upgrade first actually [03:42:51] so, if you just shut it down, the client will throw an exception [03:43:14] which is fine, the job will be reattempted [03:43:34] ah good [03:45:52] !log stopping es1004 mysqld for precise & mariadb upgrade, plus reboot [03:46:07] Logged the message, Master [03:49:28] PROBLEM - mysqld processes on es1004 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [03:49:31] the S[1-7] wikiadmin connections are usually faster to die off after a depool. ES wikiadmin jobs have more persistent connections [03:49:39] oh, icinga, forgot about you [04:02:58] TimStarling: externalLoads db-eqiad.php:315 es1005; are ES cluster masters identified by mediawiki solely by position in that array, just as for sectionLoads ? [04:04:10] yes [04:04:23] thanks [04:12:13] (03PS1) 10Springle: switch es1004 to mariadb [operations/puppet] - 10https://gerrit.wikimedia.org/r/98756 [04:14:05] TimStarling, maybe you know - (i will ask aaron tmrw otherwise) - have you set up flaggedrevs ext ? does it require complex db schema changes? [04:14:23] i need to set it up on meta [04:19:14] just extra tables [04:20:41] you can use sql.php [04:21:50] yurik: Did you ask Meta-Wiki if it wanted FlaggedRevs? [04:22:28] Elsie, no, but i am only enabling it on zero config namespace - hence it won't change anything for anyone [04:22:39] I'm not sure FlaggedRevs is still supported. [04:22:53] ??? i thought half of the world's wikis run on it [04:23:04] at least the bigger ones i think [04:23:06] de, ru [04:23:09] (03PS2) 10Springle: switch es1004 to mariadb [operations/puppet] - 10https://gerrit.wikimedia.org/r/98756 [04:23:12] Do the Germans still use it? [04:23:17] i think so [04:23:21] Yes. [04:23:23] and pt [04:23:23] (03CR) 10Springle: [C: 032] switch es1004 to mariadb [operations/puppet] - 10https://gerrit.wikimedia.org/r/98756 (owner: 10Springle) [04:23:26] en.wiki tried it a few times. [04:23:40] I don't think anyone ever liked it very much. [04:23:42] en uses it under the name 'Pending changes' [04:24:08] It was slow, very heavy on config flags, and created backlogs. [04:24:30] Who's maintaining it these days? [04:25:46] https://noc.wikimedia.org/conf/flaggedrevs.dblist [04:26:53] That's about 50 out of 800 wikis (12.5%?). [04:52:22] PROBLEM - MySQL Replication Heartbeat on db1046 is CRITICAL: CRIT replication delay 301 seconds [04:52:42] PROBLEM - MySQL Slave Delay on db1046 is CRITICAL: CRIT replication delay 303 seconds [05:11:13] ori-l, fonud something interesting ... [05:12:15] <^d> 2200 pages/s? Much more reasonable indexing speed. [05:12:24] <^d> Jobqueue was totally the right way to go here :) [05:13:39] PROBLEM - Puppet freshness on cp3012 is CRITICAL: Last successful Puppet run was Mon 02 Dec 2013 05:10:45 PM UTC [05:17:39] PROBLEM - Puppet freshness on cp3011 is CRITICAL: Last successful Puppet run was Mon 02 Dec 2013 05:15:19 PM UTC [05:21:31] ok, i think i found a techy troll of some sort - they must have a script of some sort to constantly ping the same page (in user namespace), and that page has so many links, it crashes zero [05:30:12] (03PS1) 10Springle: repool es1004 after upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98758 [05:30:42] (03CR) 10Springle: [C: 032] repool es1004 after upgrade [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98758 (owner: 10Springle) [05:32:07] !log springle synchronized wmf-config/db-eqiad.php 'repool es1004 after upgrade, max_connections lowered during warm up' [05:32:21] Logged the message, Master [05:41:58] ACKNOWLEDGEMENT - RAID on db47 is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded) Sean Pringle Imminent decommission. [05:46:04] classy! [05:56:35] (03CR) 10Dzahn: "since this is going to be on a misc host and SSL has been mentioned, you'll need :443 in the Apache config, and the cert/key setup in pupp" [operations/puppet] - 10https://gerrit.wikimedia.org/r/98740 (owner: 10BryanDavis) [06:00:57] (03CR) 10Ori.livneh: "@Dzahn: Paravoid recommended putting this behind misc-varnish in eqiad, which would get it free SSL termination, too" [operations/puppet] - 10https://gerrit.wikimedia.org/r/98740 (owner: 10BryanDavis) [06:05:04] (03CR) 10Dzahn: "ok, gotcha. well then.. i was just about to comment that of the other roles on this same node, planet has it's own wildcard cert and insta" [operations/puppet] - 10https://gerrit.wikimedia.org/r/98740 (owner: 10BryanDavis) [06:06:32] ori-l: looks like etherpad relies on star.wm just being there [06:07:04] at some point we wanted to get rid of it altogether and have more separate certs.. i dunno now [06:07:22] if it's all behind misc varnish ..ok [06:07:44] ..away [06:07:45] i don't know much about the benefits and tradeoffs of granular certificates [06:08:25] first thing, it uses a cert, but it's not puppetized that it should be there [06:08:30] but having a few closely-monitored entry points is nice; makes the attack surface smaller [06:08:36] and the other service doesnt care, because it has its own [06:11:59] and since we got from different vendors we also need to keep this updated or we create broken chains https://gerrit.wikimedia.org/r/#/c/97633/1/manifests/certs.pp [06:12:07] cya latr.. late.. away for real [06:14:23] mutante: good night [06:28:10] PROBLEM - udp2log log age for lucene on oxygen is CRITICAL: CRITICAL: log files /a/log/lucene/lucene.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [06:29:10] RECOVERY - udp2log log age for lucene on oxygen is OK: OK: all log files active [06:53:31] hello [06:57:33] morning, paravoid [06:59:10] heh, of course ori-l is still here :-) [06:59:44] i was off for a while, made salad with my son and put him to bed [07:10:55] (03PS1) 10Faidon Liambotis: protoproxy: set proxy_read_timeout to 180s [operations/puppet] - 10https://gerrit.wikimedia.org/r/98761 [07:11:26] sad [07:15:18] (03CR) 10Faidon Liambotis: [C: 032] protoproxy: set proxy_read_timeout to 180s [operations/puppet] - 10https://gerrit.wikimedia.org/r/98761 (owner: 10Faidon Liambotis) [07:15:19] (03PS1) 10Faidon Liambotis: Kill stray nginx configs for thumbs & old mobile [operations/puppet] - 10https://gerrit.wikimedia.org/r/98762 [07:16:30] (03CR) 10Faidon Liambotis: [C: 032] Kill stray nginx configs for thumbs & old mobile [operations/puppet] - 10https://gerrit.wikimedia.org/r/98762 (owner: 10Faidon Liambotis) [08:14:15] PROBLEM - Puppet freshness on cp3012 is CRITICAL: Last successful Puppet run was Mon 02 Dec 2013 05:10:45 PM UTC [08:16:15] !log nikerabbit synchronized php-1.23wmf5/extensions/Translate/ [08:16:32] Logged the message, Master [08:18:15] PROBLEM - Puppet freshness on cp3011 is CRITICAL: Last successful Puppet run was Mon 02 Dec 2013 05:15:19 PM UTC [08:27:11] !log nikerabbit synchronized php-1.23wmf4/extensions/Translate/ [08:27:26] Logged the message, Master [09:12:50] hello [09:16:05] (03PS4) 10Spage: Enable Flow discussions on a few test wiki pages [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94106 [09:21:11] (03CR) 10Spage: "Comments addressed in PS4, which is rebased with labs improvements." (032 comments) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94106 (owner: 10Spage) [09:21:18] (03CR) 10Ryan Lane: "The only thing that could possibly be in the iptables rules on virt0 is the opendj rules that are added for 389/636. Everything except the" [operations/puppet] - 10https://gerrit.wikimedia.org/r/98307 (owner: 10Faidon Liambotis) [09:23:52] who is the magic guy for vagrant again? [09:27:32] p858snake|l: ori-l [09:28:01] he is SF based so 'hopefully' sleeping [09:35:25] (03PS1) 10Hashar: deployment: integration/slave-scripts has submodules [operations/puppet] - 10https://gerrit.wikimedia.org/r/98778 [09:43:22] (03CR) 10ArielGlenn: [C: 032] beta: let jenkins-deploy restart Parsoid [operations/puppet] - 10https://gerrit.wikimedia.org/r/98685 (owner: 10Hashar) [09:52:20] (03CR) 10Hashar: "sudo command validated on the beta cluster. Thanks!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/98685 (owner: 10Hashar) [10:04:37] RECOVERY - MySQL Slave Delay on db1046 is OK: OK replication delay 0 seconds [10:05:08] RECOVERY - MySQL Replication Heartbeat on db1046 is OK: OK replication delay -0 seconds [10:08:25] (03CR) 10Ryan Lane: [C: 032] deployment: integration/slave-scripts has submodules [operations/puppet] - 10https://gerrit.wikimedia.org/r/98778 (owner: 10Hashar) [10:18:59] back [10:19:08] Ryan_Lane: do you still have merge rights on the puppetmaster ? [10:19:31] already merged it there [10:19:35] \O/ [10:19:43] ehm, s1 is lagging [10:20:04] MaxSem: db1046 had some lag apparently, might be related ? [10:20:16] hashar, all slaves ATM [10:23:39] I wonder if the cause was me with https://en.wikipedia.org/w/index.php?title=User:Rich_Farmbrough/WP_v0.8_full_index&diff=584344722&oldid=491316699 :P [10:24:09] all normal now [10:30:15] apache@deployment-jobrunner08:~$ bash /usr/local/bin/jobs-loop.sh [10:30:16] Creating default and immediate job runner pipelines [10:30:16] Warning: Invalid [10:30:19] how helpful is that [10:30:27] that is equivalent to "ERROR3 [10:42:38] (03PS1) 10Hashar: beta: define $wmgParserCacheDBs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98793 [10:43:34] (03CR) 10Hashar: [C: 032] beta: define $wmgParserCacheDBs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98793 (owner: 10Hashar) [10:43:42] (03Merged) 10jenkins-bot: beta: define $wmgParserCacheDBs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/98793 (owner: 10Hashar) [11:15:05] PROBLEM - Puppet freshness on cp3012 is CRITICAL: Last successful Puppet run was Mon 02 Dec 2013 05:10:45 PM UTC [11:19:05] PROBLEM - Puppet freshness on cp3011 is CRITICAL: Last successful Puppet run was Mon 02 Dec 2013 05:15:19 PM UTC [11:21:35] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [11:33:33] sigh [11:33:41] mobile esams is just cursed [11:36:20] PROBLEM - Varnish HTTP mobile-backend on cp3012 is CRITICAL: Connection refused [11:37:02] !log stopping varnish backend on cp3012, crashing every 30s [11:37:19] Logged the message, Master [11:37:51] bblack: ping? [11:38:55] should be a bit early for him? [11:39:05] yes, but you never know [11:49:02] (03CR) 10Akosiaris: [C: 032] reset repo by merging in tag 'v0.7.1' from upstream [operations/debs/jenkins-debian-glue] - 10https://gerrit.wikimedia.org/r/95422 (owner: 10Hashar) [11:51:25] (03CR) 10Akosiaris: [C: 032] bump wmf package to 0.7.1-6-gf618f4d [operations/debs/jenkins-debian-glue] - 10https://gerrit.wikimedia.org/r/95424 (owner: 10Hashar) [11:51:49] akosiaris: that one is a bit messy not sure whether it make sense to you [11:52:07] akosiaris: I have crafted the version in the change log using git-describe [11:53:46] hashar: yes I noticed [11:54:00] normally i 'd object to -githash [11:54:19] but the version is good enough to warrant an easy upgrade to v0.7.2 [11:54:26] when it is out [11:55:30] that was my idea [11:55:50] hashar: in general you want to avoid git hashes in version [11:56:10] because the next one might be alpabetically lower than the previous one hence no upgrade [11:57:08] but git describe list the number of commits since last tag [11:57:17] so 0.7.1-6-xxx will become 0.7.1-7-yyyy [11:57:36] I am not sure how to have git build package generate that [11:57:39] which is why I did not object :-) [11:57:45] \O/ [12:01:12] today I learned, init scripts need to be run as root .. [12:01:36] :-) [12:04:24] (03PS1) 10Hashar: beta: parsoid init script needs root [operations/puppet] - 10https://gerrit.wikimedia.org/r/98798 [12:10:52] hmm [12:14:31] bah now it works grmblblbl [12:16:51] (03Abandoned) 10Hashar: beta: parsoid init script needs root [operations/puppet] - 10https://gerrit.wikimedia.org/r/98798 (owner: 10Hashar) [12:33:58] i hate it [12:48:14] dare I ask? [12:48:57] trying to figure out why stderr is stripped out when doing " ssh sudo ..." [12:49:04] !log uploaded new jenkins-debian-glue on apt.wikimedia.org, version 0.7.1-6-gf618f4d [12:49:17] Logged the message, Master [12:51:02] PROBLEM - Host mw72 is DOWN: PING CRITICAL - Packet loss = 100% [12:52:22] RECOVERY - Host mw72 is UP: PING OK - Packet loss = 0%, RTA = 35.39 ms [12:59:46] anyone happen to reboot mw72 a couple minutes ago? [13:07:02] PROBLEM - NTP on mw72 is CRITICAL: NTP CRITICAL: Offset unknown [13:11:02] RECOVERY - NTP on mw72 is OK: NTP OK: Offset 0.0008229017258 secs [13:34:03] RECOVERY - Varnish HTTP mobile-backend on cp3012 is OK: HTTP OK: HTTP/1.1 200 OK - 189 bytes in 0.193 second response time [13:34:22] RECOVERY - Puppet freshness on cp3012 is OK: puppet ran at Tue Dec 3 13:34:16 UTC 2013 [13:34:22] RECOVERY - Puppet freshness on cp3011 is OK: puppet ran at Tue Dec 3 13:34:21 UTC 2013 [13:39:02] PROBLEM - Varnish HTTP mobile-backend on cp3012 is CRITICAL: Connection refused [13:42:03] RECOVERY - Varnish HTTP mobile-backend on cp3012 is OK: HTTP OK: HTTP/1.1 200 OK - 188 bytes in 0.194 second response time [13:58:20] (03Restored) 10Hashar: beta: parsoid init script needs root [operations/puppet] - 10https://gerrit.wikimedia.org/r/98798 (owner: 10Hashar) [13:58:35] apergos: mind merging in change above ? Some sudo policy for beta was wrong [13:58:51] the part would let jenkins-deploy use the init script as root instead of 'parsoid' user [13:59:21] i hate the parsoid shell runner … sudo -E -u parsoid nohup node /var/lib/parsoid/Parsoid/js/api/server.js > /dev/null 2>&1 & [13:59:22] :( [14:00:43] wunnerful [14:00:57] yeah I should have thought of that since it's an init script [14:01:01] anyways [14:01:18] I could not catch the std err output because the init script just run a shell that goes to background (and fails later on) [14:01:28] they parsoid team has requested help with writing an init script a number of times [14:01:36] requested from ops, that is [14:01:39] (03CR) 10ArielGlenn: [C: 032] beta: parsoid init script needs root [operations/puppet] - 10https://gerrit.wikimedia.org/r/98798 (owner: 10Hashar) [14:01:42] and we haven't responded to that request [14:01:42] !log depooling cp3012.esams temporarily to fix various ugly issues [14:01:59] Logged the message, Master [14:02:19] apergos: feel free if you have any spare cycles ;) [14:02:39] not yet but I'll add it to the todos [14:04:33] PROBLEM - Varnish HTTP mobile-frontend on cp3012 is CRITICAL: Connection refused [14:05:02] PROBLEM - Varnish HTTP mobile-backend on cp3012 is CRITICAL: Connection refused [14:06:20] close to 1,5 hour without power.... [14:06:38] at some point the laptop's battery is going to die.... [14:06:47] !log rebooting cp3012.esams [14:07:02] Logged the message, Master [14:08:40] PROBLEM - Host cp3012 is DOWN: PING CRITICAL - Packet loss = 100% [14:12:15] apergos: thank you :) [14:12:21] working? [14:12:26] (03PS2) 10Hashar: beta: autoupdate should restart parsoid [operations/puppet] - 10https://gerrit.wikimedia.org/r/98007 [14:12:41] (03PS3) 10Hashar: beta: autoupdate should restart parsoid [operations/puppet] - 10https://gerrit.wikimedia.org/r/98007 [14:13:22] apergos: yeah that is working :-] [14:13:34] the way we start parsoid is definitely not nice [14:13:50] RECOVERY - Host cp3012 is UP: PING OK - Packet loss = 0%, RTA = 96.20 ms [14:14:01] I update the python script that updates beta : https://gerrit.wikimedia.org/r/#/c/98007/ tested it and it is working fine [14:14:10] RECOVERY - Varnish HTTP mobile-backend on cp3012 is OK: HTTP OK: HTTP/1.1 200 OK - 188 bytes in 0.193 second response time [14:14:13] there was a patch around actually for an upstart job [14:14:24] that would be nie [14:14:25] nice [14:14:30] RECOVERY - Varnish HTTP mobile-frontend on cp3012 is OK: HTTP OK: HTTP/1.1 200 OK - 262 bytes in 0.193 second response time [14:14:51] I need to talk to them about the onlyinclude bug though and make sure it's on someone's radar.... tonight, forgot yesterday [14:15:45] bah my code is wrong [14:15:50] !log cp3012 re-pooled [14:16:06] Logged the message, Master [14:25:40] (03PS4) 10Hashar: beta: autoupdate should restart parsoid [operations/puppet] - 10https://gerrit.wikimedia.org/r/98007 [14:27:04] apergos: could use https://gerrit.wikimedia.org/r/#/c/98007/ now :-D [14:27:06] final call! [14:27:33] you call as often as you need, that's (part of) why I'm here :-) [14:27:52] :D [14:28:05] but yeah upstart for parsoid would be nice [14:28:13] yep [14:28:17] or anything more robust than the lame /usr/bin/parsoid shell script wrapper [14:28:24] with log rotation. seriously. [14:28:42] well the workers should send their log over syslog [14:28:49] so we can aggregate them in logstash or whatever [14:29:19] they can produce a lot of crap when things are broken [14:29:20] iirc you told me it got reported to them [14:29:53] yep, but who knows what is on the top of their pile [14:30:22] no clue [14:30:43] i have some interactions with parsoid team, VE itself is more or less a blackbox [14:30:59] (I think that is intentional, with James_F protecting the devs from outside world interruptions) [14:31:18] sure [14:34:03] apergos: meanwhile I could use https://gerrit.wikimedia.org/r/#/c/98007/ :D [14:39:47] so if the status command gives you nonzero you just log it and move on? [14:40:40] you know you just relinked the same patchset twice right? [14:40:42] hashar: [14:40:47] arghgh [14:40:58] so yeah errors are not blocking, just logged [14:41:10] just wondering if 5 seconds is long enough [14:41:27] seems like [14:41:36] the status script invoke a shell script that basically does ( … ) & [14:41:39] well you can always adjust later [14:41:40] so it returns asap [14:41:54] and start-stop-daemon always returns 0 cause ( .. ) & is always a success [14:42:18] yep [14:42:31] all righty [14:42:47] (03CR) 10ArielGlenn: [C: 032] beta: autoupdate should restart parsoid [operations/puppet] - 10https://gerrit.wikimedia.org/r/98007 (owner: 10Hashar) [14:50:41] 00:00:00.561 fatal: Could not change back to '/home/jenkins-deploy/workspace/beta-code-update': Permission denied [14:50:43] oh yeah [14:50:46] I need a new carrer [14:51:43] should swipe streets with a broom maybe [14:52:36] you'd get bored of that pretty quick [14:52:44] also too much rain and cold for that [14:56:33] on the other hand I am getting fed up with software bugs [14:56:50] git log issuing a fatal because it attempts to chdir :/ [15:00:03] it just needs to be able to read, is the directory readable only by jenkins-deploy? [15:06:22] (03PS1) 10Faidon Liambotis: protoproxy: add proxy_read_timeout to IPv6 too [operations/puppet] - 10https://gerrit.wikimedia.org/r/98824 [15:07:02]