[00:02:03] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Server Error - 1703 bytes in 6.727 second response time [00:02:22] PROBLEM - Varnishkafka Delivery Errors on cp3020 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 331.533325 [00:10:02] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 209534 bytes in 7.153 second response time [00:12:02] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:12:22] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:13:22] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:15:22] RECOVERY - Varnishkafka Delivery Errors on cp3020 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:17:52] !log ori synchronized php-1.23wmf16/extensions/CentralNotice 'Update CentralNotice to tip of wmf_deploy for I7d8259fc4' [00:18:00] Logged the message, Master [00:19:42] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 08:45:44 AM UTC [00:24:46] !log ori synchronized php-1.23wmf15/extensions/CentralNotice 'Update CentralNotice to tip of wmf_deploy for I7d8259fc4' [00:24:54] Logged the message, Master [00:25:05] ^ mwalker, K4-713, greg-g [00:26:51] greg-g: around? [00:28:45] ori: Thanks for the update. [00:33:34] Getting a report of issues with bits [00:33:45] I can't repro, but it's there [00:34:19] rdwrer: Looks normal from Europe [00:34:21] Maybe just residual from the PROBLEMs above? [00:34:45] Report is from NY, US [00:34:57] jeremyb: How do things look for you? [00:36:08] ori: I don't suppose there were issues with your deploy that may be causing this [00:37:12] with what? [00:37:19] causing what, rather? [00:37:31] what 'issues with bits'? [00:37:33] There's a report about bits issues, trying to confirm now [00:37:37] JS and images not loading [00:37:40] i'll look at ganglia [00:38:06] rdwrer: which page/hostname? [00:38:18] enwiki, she says she can't log in [00:38:30] i.e. the login page won't load [00:39:32] works for me, but i'll continue looking around for an indication that something may be wrong [00:40:24] Thanks, not sure if it's just her or what [00:40:44] we had someone with similar issues in here yesterday [00:40:46] rdwrer: do you know which browser she's using? [00:40:51] but weren't able to nail down anything [00:41:10] hoo: Was it Finnegan? [00:41:27] rdwrer: No, Cyberpower... [00:41:29] Hm. [00:43:00] ori: Neither fx nor chrome work well for her [00:43:14] Something will load, but not all of it [00:43:23] rdwrer: Can you get a traceroute or ping out of her? [00:43:36] Can try [00:44:05] Or do one better [00:44:20] mh? [00:45:08] Trying to bring her in [00:46:55] Finnegan: 2014-03-04 - 16:43:23 rdwrer: Can you get a traceroute or ping out of her? [00:47:20] i can try if someone can explain how I do that? [00:47:29] Hm, Windows? [00:47:34] yep, win7 [00:47:35] Finnegan: actually -- there's something you could try first [00:47:41] you have Chrome handy, right? [00:48:05] yes. Also Opera, somewhere in the depths of my drive [00:48:17] can you browse to the login page that appears broken [00:48:36] and then use the browser's menu bar to choose View -> Developer -> JavaScript Console [00:49:32] "[blocked] The page at 'https://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Incidents' was loaded over HTTPS, but ran insecure content from 'http://en.wikipedia.org/w/index.php?action=raw&ctype=text/css&title=User:Lupin/navpopdev.css': this content should also be loaded over HTTPS." [00:50:04] (the login seems to have gone through, and it's trying to take me to ANI, which is where I was going before I couldn't get there) [00:51:14] mh [00:51:34] there's a couple "hey, this parameter is deprecated" warnings in there too [00:51:48] that's normal [00:52:04] ori: Can the https things make Chrome block other content [00:52:12] * hoo hasn't touched chrome in months [00:52:33] I mean will it stop loading like on a JS syntax error [00:53:19] I'm looking at what I think is the corresponding console in firefox now while trying to load ANI directly. Not seeing the "insecure" warning in the console, but the page is still hung with most of it loaded (only the logo is currently missing). [00:53:48] * Finnegan apologizes for being the most worthlessly non-technical bug-reporter ever [00:54:15] Finnegan: not at all! I hope we can help [00:58:30] Maybe another problem in -tech [00:58:31] Maybe the same [01:00:08] Finnegan: is it still hung? [01:00:33] did you open the console in firefox, or did it come up by itself? is there an error indicated? [01:01:17] I opened it in firefox myself. It was hung as of about a minute ago, at which point I canceled the pag eload because my fan was starting to make fan-y noises [01:01:40] no error messages that I can see. Only the yellow "deprecated" messages [01:02:28] "because my fan was starting to make fan-y noises" sounds a bit like run away javascript [01:03:33] mh, there's quite some JS there: https://en.wikipedia.org/wiki/User:Fluffernutter/common.js [01:03:43] * Finnegan blushes [01:05:51] Finnegan: Tested your JS in my account and it worked ok [01:07:50] currently "waiting for upload.wikimedia.org" according to ff [01:08:27] Finnegan: On which page? [01:08:37] ANI still. Want me to try another? [01:08:41] Or can you maybe even see the full URL of what it's trying to load [01:09:53] not that I can see. That's just the snippet from the browser's status bar [01:19:44] Finnegan: Leaving in a few minutes, next step would probably be to get a traceroute to upload.wikimedia.org [01:19:52] rdwrer: Want to guide? ;) [01:27:42] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Last successful Puppet run was Tue 25 Feb 2014 06:33:37 PM UTC [02:01:22] (03PS1) 10Chad: Give Cirrus to all Wikiquotes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/116919 [02:14:29] (03CR) 10Deskana: [C: 031] "Product manager says yes." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/116919 (owner: 10Chad) [02:16:57] Database error A database query error has occurred. This may indicate a bug in the software. Function: SpecialWhatLinksHere::showIndirectLinks Error: 0 [02:16:58] at https://en.wikipedia.org/w/index.php?title=Special%3AWhatLinksHere&target=Module%3AUnsubst&namespace=10 [02:24:11] 504 for the same url now [02:27:46] !log LocalisationUpdate completed (1.23wmf15) at 2014-03-05 02:27:46+00:00 [02:27:54] Logged the message, Master [02:54:01] !log LocalisationUpdate completed (1.23wmf16) at 2014-03-05 02:54:00+00:00 [02:54:08] Logged the message, Master [02:54:19] I wonder how long that took.... [03:20:42] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 08:45:44 AM UTC [03:37:06] !log messing with innodb compression on db1007 [03:37:14] Logged the message, Master [03:41:14] !log LocalisationUpdate ResourceLoader cache refresh completed at 2014-03-05 03:41:14+00:00 [03:41:23] Logged the message, Master [03:55:50] ok, starting on the move to trigger [03:55:53] (03CR) 10BryanDavis: Config changes for Elasticsearch (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/116498 (owner: 10Manybubbles) [03:58:21] (03CR) 10BryanDavis: [C: 031] Log length of l10nupdate to SAL and Graphite [operations/puppet] - 10https://gerrit.wikimedia.org/r/116718 (owner: 10Greg Grossmeier) [04:03:34] (03PS6) 10Ryan Lane: Deployment module changes for trebuchet-trigger [operations/puppet] - 10https://gerrit.wikimedia.org/r/110239 [04:08:00] rdwrer: sorry, was afk [04:08:09] 's okay [04:08:52] PROBLEM - MySQL Idle Transactions on db1033 is CRITICAL: CRIT longest blocking idle transaction sleeps for 604 seconds [04:09:13] PROBLEM - MySQL InnoDB on db1033 is CRITICAL: CRIT longest blocking idle transaction sleeps for 626 seconds [04:17:36] !log switching out git-deploy perl frontend for trebuchet trigger [04:17:44] Logged the message, Master [04:18:02] (03CR) 10Ryan Lane: [C: 032] Deployment module changes for trebuchet-trigger [operations/puppet] - 10https://gerrit.wikimedia.org/r/110239 (owner: 10Ryan Lane) [04:21:52] RECOVERY - MySQL Idle Transactions on db1033 is OK: OK longest blocking idle transaction sleeps for 0 seconds [04:22:13] RECOVERY - MySQL InnoDB on db1033 is OK: OK longest blocking idle transaction sleeps for 0 seconds [04:28:38] Ryan_Lane: \o/ [04:28:40] that's awesome [04:28:42] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Last successful Puppet run was Tue 25 Feb 2014 06:33:37 PM UTC [04:28:56] (03PS1) 10Ryan Lane: Adding python-gitdb dependency [operations/puppet] - 10https://gerrit.wikimedia.org/r/116924 [04:29:00] well, I still need to test ;) [04:29:23] but it should be in place soon [04:29:57] i think scap / trebuchet convergence is a lot closer now, btw [04:29:57] hm. I really need test/testrepo pointing at something [04:30:04] ori: any idea where I can point it? [04:30:08] excellent [04:30:17] what do you mean? which repo or server or? [04:30:19] I'm looking forward to having some of the code merge together :) [04:30:21] server [04:30:27] it used to point at formey, I think [04:30:41] what are the desirable characteristics for that server? [04:30:50] i dunno what your criteria are [04:30:52] it's the most basic of basic repos [04:30:55] it can be anything [04:31:16] it can be hafnium, that's an odds and ends server for metrics [04:31:23] ok, cool [04:32:07] hm. I wonder if formey is still targetable [04:32:13] if so, I'll use it for this test [04:32:22] sure [04:33:42] -_- [04:33:49] IMissing the following configuration item: deploy.repo-name [04:33:54] I forgot I changed that name [04:34:49] working for test/testrepo [04:35:02] now I need to update that damn key on all the repos [04:35:25] I guess I could have kept compatibility, but prefix-name sucks [04:38:10] (03PS1) 10Ryan Lane: Change sudo format for pillar fetching and service restarts for trigger [operations/puppet] - 10https://gerrit.wikimedia.org/r/116925 [04:43:25] ori: any repos that we can test a deploy with? [04:43:48] bleh. umask isn't being checked [04:44:15] fluoride or eventlogging would be fine. does there need to be an undeployed change? [04:44:23] i can touch a README or something if you like [04:44:59] nah, it can be a no-op change [04:46:38] damn git-python. I need to switch that out with dulwich [04:46:45] it's not reading the global config properly [04:47:09] oh well, required umask is at best only a partially useful feature [04:48:02] * Ryan_Lane opens a bug [04:48:15] fluoride would be safest [04:48:48] ok [04:57:42] and of course, I'm trying to require the wrong umask. it should be 002 [05:00:47] ah. interesting [05:01:01] vanadium has a deployment_target grain of eventlogging [05:01:09] which doesn't match fluroride [05:01:26] fluoride* [05:03:08] ori: when was the last time that was deployed? [05:03:50] I think it's target was removed from puppet [05:03:53] *its [05:04:38] (03CR) 10Ryan Lane: [C: 032] Adding python-gitdb dependency [operations/puppet] - 10https://gerrit.wikimedia.org/r/116924 (owner: 10Ryan Lane) [05:04:57] (03CR) 10Ryan Lane: [C: 032] Change sudo format for pillar fetching and service restarts for trigger [operations/puppet] - 10https://gerrit.wikimedia.org/r/116925 (owner: 10Ryan Lane) [05:05:00] PROBLEM - Puppet freshness on mw32 is CRITICAL: Last successful Puppet run was Wed 05 Mar 2014 05:00:17 AM UTC [05:05:49] (03PS1) 10Ryan Lane: Change required umask for deployment in trigger to 002 [operations/puppet] - 10https://gerrit.wikimedia.org/r/116926 [05:07:00] PROBLEM - Puppet freshness on mw32 is CRITICAL: Last successful Puppet run was Wed 05 Mar 2014 05:00:17 AM UTC [05:08:42] (03PS1) 10Ryan Lane: fluoride is no longer a target, switch to eventlogging [operations/puppet] - 10https://gerrit.wikimedia.org/r/116927 [05:09:00] PROBLEM - Puppet freshness on mw32 is CRITICAL: Last successful Puppet run was Wed 05 Mar 2014 05:00:17 AM UTC [05:10:15] rawr jenkins [05:11:00] PROBLEM - Puppet freshness on mw32 is CRITICAL: Last successful Puppet run was Wed 05 Mar 2014 05:00:17 AM UTC [05:13:00] PROBLEM - Puppet freshness on mw32 is CRITICAL: Last successful Puppet run was Wed 05 Mar 2014 05:00:17 AM UTC [05:15:00] PROBLEM - Puppet freshness on mw32 is CRITICAL: Last successful Puppet run was Wed 05 Mar 2014 05:00:17 AM UTC [05:15:10] (03CR) 10Ryan Lane: [C: 032] Change required umask for deployment in trigger to 002 [operations/puppet] - 10https://gerrit.wikimedia.org/r/116926 (owner: 10Ryan Lane) [05:15:22] (03CR) 10Ryan Lane: [C: 032] fluoride is no longer a target, switch to eventlogging [operations/puppet] - 10https://gerrit.wikimedia.org/r/116927 (owner: 10Ryan Lane) [05:17:00] PROBLEM - Puppet freshness on mw32 is CRITICAL: Last successful Puppet run was Wed 05 Mar 2014 05:00:17 AM UTC [05:19:00] PROBLEM - Puppet freshness on mw32 is CRITICAL: Last successful Puppet run was Wed 05 Mar 2014 05:00:17 AM UTC [05:20:28] that fixed fluoride, but it's being deployed one more place than before (hafnium.wikimedia.org and vanadium.eqiad.wmnet) [05:21:00] PROBLEM - Puppet freshness on mw32 is CRITICAL: Last successful Puppet run was Wed 05 Mar 2014 05:00:17 AM UTC [05:21:08] service restart is working, report sync is working, sync is working, start is working, abort is working [05:21:10] success! [05:21:22] !log finished switch from the perl git-deploy to trigger [05:21:31] Logged the message, Master [05:23:00] PROBLEM - Puppet freshness on mw32 is CRITICAL: Last successful Puppet run was Wed 05 Mar 2014 05:00:17 AM UTC [05:24:50] \o/ [05:25:00] PROBLEM - Puppet freshness on mw32 is CRITICAL: Last successful Puppet run was Wed 05 Mar 2014 05:00:17 AM UTC [05:25:02] no more shitty perl! [05:25:04] :) [05:25:20] not that my python is really amazing, but it's nicer than perl [05:27:00] PROBLEM - Puppet freshness on mw32 is CRITICAL: Last successful Puppet run was Wed 05 Mar 2014 05:00:17 AM UTC [05:28:16] I should probably do a no-op deploy to parsoid [05:29:00] PROBLEM - Puppet freshness on mw32 is CRITICAL: Last successful Puppet run was Wed 05 Mar 2014 05:00:17 AM UTC [05:30:50] RECOVERY - Puppet freshness on mw32 is OK: puppet ran at Wed Mar 5 05:30:45 UTC 2014 [05:31:25] <3 deploys are way faster now that I don't need to request pillar data a bunch of times [05:33:00] PROBLEM - Puppet freshness on mw32 is CRITICAL: Last successful Puppet run was Wed 05 Mar 2014 05:30:45 AM UTC [05:47:14] (03PS1) 10Ori.livneh: Enable cookie-based geolocation on all frontend text Varnishes [operations/puppet] - 10https://gerrit.wikimedia.org/r/116928 [06:00:28] RECOVERY - Puppet freshness on mw32 is OK: puppet ran at Wed Mar 5 06:00:23 UTC 2014 [06:21:28] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 08:45:44 AM UTC [07:29:28] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Last successful Puppet run was Tue 25 Feb 2014 06:33:37 PM UTC [08:16:25] wait [08:16:29] dickson.freenode.net split? [08:16:45] How come? [08:18:13] (03CR) 10Alexandros Kosiaris: [C: 032] contint: do not cache api/json calls [operations/puppet] - 10https://gerrit.wikimedia.org/r/116748 (owner: 10Hashar) [08:19:57] apergos: dickson.freenode.net split? [08:19:58] oO [08:20:03] weird.. [08:20:12] hello, yes likely so [08:20:18] Hmm [08:20:22] Did RC go down also? [08:20:32] I see it didn't lol [08:20:33] rc? [08:20:39] irc.wikimedia.org [08:20:40] oh [08:20:42] Different server? [08:20:48] different server and system [08:20:57] :) [08:21:45] I was trying to troubleshoot my IRC bot and saw "Max SendQ exceeded." on the RC server.. [08:21:46] :S [08:21:59] With ngrep :D [08:22:54] nice :-D [08:27:22] Yeah, but of course it doesn't work on SSL connections ;) [08:27:38] I'm mainly connected to SSL-enabled IRC servers. [08:27:43] (Except my bot) [08:29:53] if dickson splits it's probably freenode doing maintenance on it [08:30:01] yup, when we cut out (some of) the ability of govts to snoop, we also cut out (some of) our troubleshooting ability [09:22:28] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 08:45:44 AM UTC [09:49:27] (03PS4) 10Ori.livneh: reprepro: import Facebook's HHVM pkgs, for Labs / Vagrant usage only [operations/puppet] - 10https://gerrit.wikimedia.org/r/112314 [09:50:31] hey [09:50:35] I was just amending that [09:52:14] (03PS5) 10Faidon Liambotis: reprepro: import Facebook's HHVM pkgs, for Labs / Vagrant usage only [operations/puppet] - 10https://gerrit.wikimedia.org/r/112314 (owner: 10Ori.livneh) [09:52:53] (03CR) 10Faidon Liambotis: [C: 032] "Persuaded to merge this, with the condition of never reaching production in this state." [operations/puppet] - 10https://gerrit.wikimedia.org/r/112314 (owner: 10Ori.livneh) [09:53:03] (03CR) 10Faidon Liambotis: [V: 032] reprepro: import Facebook's HHVM pkgs, for Labs / Vagrant usage only [operations/puppet] - 10https://gerrit.wikimedia.org/r/112314 (owner: 10Ori.livneh) [09:54:31] paravoid: hi! Is the hhvm package installable on Precise? :-] [09:54:52] yes, the hhvm "package" is installable on precise, only for testing purposes [09:55:08] awesome [09:55:26] meh [09:55:36] This way I will be able to add a Jenkins job to run mw/core test suite :-] [10:02:17] hashar: it can't / shouldn't run on the production jenkins server [10:02:39] that was the condition for merging it, as the packages have some rather severe problems [10:02:45] (03PS1) 10Matanya: sockpuppet: remove leftovers [operations/puppet] - 10https://gerrit.wikimedia.org/r/116936 [10:03:11] !log Jenkins: gallium overloaded with a few java threads taking 100% CPU starving the box :( [10:03:19] Logged the message, Master [10:03:25] ori: will get it running on a labs instance [10:03:30] * ori nods [10:03:38] paravoid: thanks for the merge, btw! [10:03:49] did you see the diff? [10:03:51] there were two errors [10:04:00] * ori looks [10:04:18] first, they were shipping InRelease, but you had GetInRelease: no [10:04:20] that's minor [10:04:25] hashar: 3rd time in 24h ... [10:04:38] the more important one is that you had Suite: stable and they have no such suite in the repo [10:04:40] matanya: yeah I am well aware of it :/ [10:05:00] matanya: I gotta drop history from Jenkins builds I guess [10:05:16] paravoid: my mistake, sorry. I wasn't sure how to test it. [10:05:26] hashar: you can store it elsewhere [10:05:35] not without engineering something new [10:05:53] we could put the logs / files in a swift volume [10:06:02] no, I'm just telling you so that you know for next time :) [10:06:08] * ori nods [10:06:19] Suite: NNN maps to http://dl.hhvm.com/ubuntu/dists/NNN [10:06:41] well, hashar an action should be taken :D [10:06:51] could we have a 'testing' component that isn't enabled on production? [10:07:04] that would be nice, to be able to add stuff for vagrant, etc. [10:07:23] meh, maintaining more components is kind of a pain, but if you have multiple use cases we should consider it [10:08:02] there are a few things vagrant provisions via PPAs and other distribution channels [10:08:13] xdebug, phpsh, hhvm (until now) [10:08:16] maybe one or two more [10:08:44] phpsh sounds like something that could be useful in prod too? [10:09:00] no way [10:09:08] i wish i never started using it [10:09:21] i now maintain the package in pypi, heh [10:09:41] i'd replace it with something like boris if i were starting over, but i've been lazy [10:09:53] it's a really, really awful hack [10:10:36] see readme: https://github.com/atdt/phpsh [10:10:47] "If you're not an existing user, stop. There are better options out there. " [10:11:01] lol [10:12:12] Downloads (All Versions): 38 downloads in the last day [10:13:40] paravoid: btw, the geo_cookie stuff is running on cp1055 and cp1066, no issues and no measurable negative impact on request latency afaict [10:14:10] so bblack and i were going to push it out to the remaining varnishes today/tomorrow [10:14:23] cool [10:16:26] (03CR) 10Matanya: [C: 031] "a followup patch: https://gerrit.wikimedia.org/r/#/c/116936/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/115527 (owner: 10Dzahn) [10:18:15] (03CR) 10TTO: "I like section collapsing for Wiktionary. (Obviously I don't like it for pages with only one section, but for more than one it is useful.)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114634 (owner: 10MaxSem) [10:29:52] (03PS2) 10Matanya: sockpuppet: remove leftovers [operations/puppet] - 10https://gerrit.wikimedia.org/r/116936 [10:30:28] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Last successful Puppet run was Tue 25 Feb 2014 06:33:37 PM UTC [10:31:20] (03CR) 10MaxSem: "Looks like consensus on the bug is different." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114634 (owner: 10MaxSem) [10:32:20] (03PS1) 10Ori.livneh: Update Schema:Echo revision to 7731316 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/116937 [10:34:02] (03CR) 10Ori.livneh: [C: 032] Update Schema:Echo revision to 7731316 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/116937 (owner: 10Ori.livneh) [10:34:14] hey, do we have any logs that can help us to figure out what's going on with https://bugzilla.wikimedia.org/show_bug.cgi?id=62245 ? [10:34:16] (03Merged) 10jenkins-bot: Update Schema:Echo revision to 7731316 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/116937 (owner: 10Ori.livneh) [10:34:51] (03PS1) 10Matanya: dhcpd: remove Sun-X4150 related configs [operations/puppet] - 10https://gerrit.wikimedia.org/r/116938 [10:36:59] MaxSem: $ curl -Is -A "DoCoMo/2.0 SH03A" http://ja.wikipedia.org/wiki/Main_page | grep ^Location [10:36:59] Location: http://ja.m.wikipedia.org/wiki/Main_page [10:38:22] however, phone users see a 502 [10:38:28] wait [10:39:00] early November - isn't it when we switched text from Squid to Varnish? [10:40:03] (03PS1) 10Hashar: contint: deny access to Jenkins build history [operations/puppet] - 10https://gerrit.wikimedia.org/r/116939 [10:40:30] yes, it's mentioned in https://www.mediawiki.org/wiki/Scrum_of_scrums/2013-11-12 [10:41:21] mark & paravoid, ^^ [10:41:34] no DoCoMo matches in our 5xx.logs [10:41:49] the user-agent strings in the bug report seem incomplete, too [10:42:32] (03CR) 10Matanya: [C: 031] contint: deny access to Jenkins build history [operations/puppet] - 10https://gerrit.wikimedia.org/r/116939 (owner: 10Hashar) [10:43:09] ori, nope - these a typical docomo UAs, e.g. http://www.tera-wurfl.com/explore/?action=wurfl_id&id=docomo_sh_03a_ver1 [10:43:45] https://productforums.google.com/forum/#!msg/webmasters/dBD59HzcRWE/YJIGcCrowUMJ [10:46:08] seems like a pretty weird setup [10:46:37] https://www.nttdocomo.co.jp/english/service/information/fullbrowser/ says "Part or all of some sites may not be viewable depending on the site." [10:46:38] hmmmmmm [10:46:46] they have an emulator on their site [10:47:32] we see a 502 message in Japanese - apparently, there's some intermediate proxy emitting it? [10:49:16] yeah, i think it's some sort of proxy that strips content that the phone isn't able to display [10:49:48] you may be causing it to freak out by using modern html [10:50:10] nope [10:50:23] we're not emitting html on the redirecth [10:51:14] hm, yeah, and they're able to access the site directly [10:57:38] ori, where did you see the emulator? I've found https://www.nttdocomo.co.jp/english/service/developer/make/content/myface/emulator/index.html but it's not what we want [10:58:01] MaxSem: https://www.nttdocomo.co.jp/english/service/developer/make/content/browser/html/tool2/download/ [10:58:20] thanks [11:01:15] akosiaris: can you please merge : https://gerrit.wikimedia.org/r/#/c/116938/ ? [11:06:32] (03CR) 10Ori.livneh: [C: 032] contint: deny access to Jenkins build history [operations/puppet] - 10https://gerrit.wikimedia.org/r/116939 (owner: 10Hashar) [11:07:17] ori: thx [11:08:12] sigh [11:08:13] !log restarting Jenkins which has starving threads [11:08:21] Logged the message, Master [11:08:30] and of course, the emulator doesn't exhibit that bug [11:11:27] !log ori updated /a/common to {{Gerrit|I5277d2451}}: Update Schema:Echo revision to 7731316 [11:11:36] Logged the message, Master [11:12:28] PROBLEM - Host mw27 is DOWN: PING CRITICAL - Packet loss = 100% [11:12:37] !log jenkins back [11:12:45] Logged the message, Master [11:13:28] RECOVERY - Host mw27 is UP: PING OK - Packet loss = 0%, RTA = 35.38 ms [11:19:41] (03PS1) 10Hashar: contint: Icinga now notify on contint related stuff [operations/puppet] - 10https://gerrit.wikimedia.org/r/116944 [11:21:38] (03PS1) 10Matanya: icinga: update check_mysql-replication to v 0.2.6 [operations/puppet] - 10https://gerrit.wikimedia.org/r/116945 [11:24:32] paravoid, did Squid emit some HTML on redirects? [11:30:39] is ken still with the wmf? [11:31:10] no [11:36:00] paravoid: any reason I can't see the hhvm package on a labs instance though it got added to reprepro ( https://gerrit.wikimedia.org/r/#/c/112314/ ) [11:38:25] try again [11:41:24] paravoid: \O/ [11:44:53] (03PS1) 10Hashar: contint: install hhvm on labs slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/116947 [11:45:56] paravoid: would love if you can security review the patch above :) [11:47:27] (03PS1) 10Matanya: icinga: remove ksnider from contact groups [operations/puppet] - 10https://gerrit.wikimedia.org/r/116948 [12:23:28] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Fri 28 Feb 2014 08:45:44 AM UTC [12:49:50] (03PS1) 10Matanya: php: remove hardy setting, no more hardy app servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/116953 [12:56:11] (03CR) 10Faidon Liambotis: [C: 032] icinga: remove ksnider from contact groups [operations/puppet] - 10https://gerrit.wikimedia.org/r/116948 (owner: 10Matanya) [12:56:21] thanks [12:59:29] paravoid: are you going to be in zurich hackathon? [12:59:49] dunno yet [13:01:26] i'm thinking of it. but i'm only intersted of ops will be there [13:01:33] *if [13:02:59] akosiaris & mark will be there [13:03:15] and andrewb/marc-andre [13:03:56] hmm, the main issue is Saturday [13:04:05] no i won't be [13:04:12] but paravoid will be there :P [13:04:14] you won't? [13:04:19] no [13:04:20] i didn't sign up [13:04:54] btw, you too, i would like to have access to ops-l if possible [13:04:58] *two [13:05:32] hm, I'm not sure if we've ever done this before [13:05:35] do you have an NDA? [13:05:45] I like the way matanya ask his questions. [13:05:56] yes paravoid [13:06:13] ok [13:06:21] Instead of "Wouldn't it be too much of an inconvenience if I could get access to ops-l, please", he says "I would like to have access to ops-l" [13:06:35] * odder pats matanya on the back [13:06:40] I don't mind [13:06:43] I like people being direct :) [13:06:51] my point precisely :-) [13:07:13] i'm not british :) [13:07:38] do you mind submitting the form at https://lists.wikimedia.org/mailman/listinfo/ops ? [13:07:44] I can't promise we'll accept it [13:07:54] but it should at least remind us to have this discussion [13:08:33] (not about you, about non-staff subscriptions in general, I'm not sure if there's a precedent for this, although I'd be in favor) [13:10:13] done paravoid thanks for honsty :) [13:10:31] why all the vowles aren't typed today? [13:12:06] You might have switched your keyboard to Hebrew by mistake, I think [13:12:09] :-DD [13:29:08] (03CR) 10Manybubbles: Config changes for Elasticsearch (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/116498 (owner: 10Manybubbles) [13:29:11] (03PS3) 10Manybubbles: Config changes for Elasticsearch [operations/puppet] - 10https://gerrit.wikimedia.org/r/116498 [13:30:44] PROBLEM - Puppet freshness on labstore4 is CRITICAL: Last successful Puppet run was Tue 25 Feb 2014 06:33:37 PM UTC [13:39:34] PROBLEM - Host labstore4 is DOWN: PING CRITICAL - Packet loss = 100% [13:42:38] :/ [13:47:14] RECOVERY - Host labstore4 is UP: PING OK - Packet loss = 0%, RTA = 35.40 ms [14:08:16] (03CR) 10Alexandros Kosiaris: [C: 04-1] "The garbage removal is cool, I would rather we did not delete however the dhcpd.conf stanza, even if the file is empty (it exists btw in t" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/116938 (owner: 10Matanya) [14:11:32] (03PS2) 10Matanya: dhcpd: remove Sun-X4150 related configs [operations/puppet] - 10https://gerrit.wikimedia.org/r/116938 [14:11:42] done akosiaris [14:12:22] (03CR) 10Alexandros Kosiaris: [C: 032] dhcpd: remove Sun-X4150 related configs [operations/puppet] - 10https://gerrit.wikimedia.org/r/116938 (owner: 10Matanya) [14:12:36] (03CR) 10Alexandros Kosiaris: [V: 032] dhcpd: remove Sun-X4150 related configs [operations/puppet] - 10https://gerrit.wikimedia.org/r/116938 (owner: 10Matanya) [14:13:47] matanya: so my catalog-differ is finally ready. Your @var changes are now officially my guinea pigs, you should see them being merged (hopefully) one by one [14:14:05] yay! :) thanks [14:14:16] is the source available ? [14:15:13] not yet. It will be though, I intend to write an installation procedure and some docs first [14:20:05] PROBLEM - Disk space on virt10 is CRITICAL: DISK CRITICAL - free space: /var/lib/nova/instances 44146 MB (3% inode=99%): [14:22:45] (03PS4) 10Manybubbles: Config changes for Elasticsearch [operations/puppet] - 10https://gerrit.wikimedia.org/r/116498 [14:25:38] (03CR) 10Ottomata: "Just looked at this again, but wasn't sure if you were ready for review since some of the comments weren't acted on. Lemme know when!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/116498 (owner: 10Manybubbles) [14:27:24] (03PS2) 10Ottomata: Changing replica_lag_max_messages and replica_lag_time_max_ms [operations/puppet] - 10https://gerrit.wikimedia.org/r/116645 [14:27:31] (03CR) 10Ottomata: [C: 032 V: 032] Changing replica_lag_max_messages and replica_lag_time_max_ms [operations/puppet] - 10https://gerrit.wikimedia.org/r/116645 (owner: 10Ottomata) [14:29:52] (03CR) 10Manybubbles: "I think we're ok with applying this as is. We'll start looking at rack awareness vs forced awareness next week." [operations/puppet] - 10https://gerrit.wikimedia.org/r/116498 (owner: 10Manybubbles) [14:31:01] !log starting controlled shutdown of kafka broker an21 to reload new replica.lag settings [14:31:11] Logged the message, Master [14:34:34] PROBLEM - Kafka Broker Under Replicated Partitions on analytics1022 is CRITICAL: kafka.server.ReplicaManager.UnderReplicatedPartitions.Value CRITICAL: 10.0 [14:35:05] (03PS1) 10Matanya: ganglia: remove lookupvar and replace with top scope @ var [operations/puppet] - 10https://gerrit.wikimedia.org/r/116973 [14:36:10] (03CR) 10Ottomata: "Aye, ok. I know my comments were nitpicky, but you wanted to fix some of them, right? strings vs regexes, undefs instead of false, etc?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/116498 (owner: 10Manybubbles) [14:37:32] (03CR) 10Manybubbles: "WTF I thought I fixed them...." [operations/puppet] - 10https://gerrit.wikimedia.org/r/116498 (owner: 10Manybubbles) [14:39:54] !log starting controlled shutdown (and restart) of kafka broker analytics1022 to bring in new replica.lag settings [14:40:02] Logged the message, Master [14:40:03] (03PS5) 10Manybubbles: Config changes for Elasticsearch [operations/puppet] - 10https://gerrit.wikimedia.org/r/116498 [14:40:05]