[00:32:09] PROBLEM - Puppet freshness on db1029 is CRITICAL: Puppet has not run in the last 10 hours [01:41:58] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 295 seconds [01:42:52] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 262 seconds [01:49:10] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 636s [01:50:58] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 12 seconds [01:53:22] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 17 seconds [01:53:40] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 6s [02:26:49] * jeremyb waves domas. nice rain? [02:32:37] yeh [02:47:10] PROBLEM - Puppet freshness on mw1102 is CRITICAL: Puppet has not run in the last 10 hours [03:01:07] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [03:39:44] * jeremyb waves Jasper_Deng ;) [03:40:18] * Jasper_Deng waves back [03:42:47] thanks for the bits :P [03:43:24] np [04:01:09] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [05:12:57] PROBLEM - Puppet freshness on ms3 is CRITICAL: Puppet has not run in the last 10 hours [05:23:54] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [05:33:57] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [05:48:57] PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours [06:33:05] is there an issue tracker where i can make tickets on ways to clean up the puppet repo? [07:00:09] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [07:23:36] ACKNOWLEDGEMENT - Host sq36 is DOWN: PING CRITICAL - Packet loss = 100% daniel_zahn RT-3254, no carrier [07:32:24] RECOVERY - Host owa3 is UP: PING WARNING - Packet loss = 66%, RTA = 0.55 ms [07:35:13] !log powercycled downed owa3, installed kernel upgrades on owa1-3 (but they are also waiting to be repurposed, RT-2511)exit [07:35:22] Logged the message, Master [07:36:36] PROBLEM - Host owa3 is DOWN: PING CRITICAL - Packet loss = 100% [07:37:39] RECOVERY - Host owa3 is UP: PING OK - Packet loss = 0%, RTA = 0.53 ms [07:43:12] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [07:49:48] PROBLEM - Host owa2 is DOWN: PING CRITICAL - Packet loss = 100% [07:51:00] RECOVERY - Host owa2 is UP: PING OK - Packet loss = 0%, RTA = 0.38 ms [07:57:41] bonjour [07:59:46] !log srv266 - package/kernel upgrades [07:59:53] Logged the message, Master [08:00:01] Guten Tag mutante [08:00:15] I need to relearn German one day :D [08:00:38] du solltest aber schnell! [08:01:06] RECOVERY - Puppet freshness on owa2 is OK: puppet ran at Wed Jul 11 08:00:40 UTC 2012 [08:01:43] heh, http://en.wiktionary.org/wiki/Category:German_phrases [08:01:58] hashar: hi, so i saw something in gerrit, re apache config [08:02:15] hashar: but i already merged that ,it was JeremyB's change [08:02:23] ohhh [08:02:27] what are you changing here: [08:02:28] that / something ? :-D [08:02:35] https://gerrit.wikimedia.org/r/#/c/15161/1/redirects.conf [08:02:50] apparently you did some local changes on fenari which were committed to svn only [08:02:58] I did some git magic to craft git commit [08:03:00] https://gerrit.wikimedia.org/r/#/c/14509/1 [08:03:03] and published them back in Gerrit [08:03:08] <-- that is the one i merged in git [08:03:19] then i just also did that in svn to push it out [08:03:32] but also the gerrit merge, it just wasnt started by me [08:03:45] hmm [08:03:51] 15161 I it is a commit I made [08:04:09] I manually added you as the author since that was in svn under your name :D [08:04:20] yeah, what does it actually change in redirects.conf [08:04:30] something like: git commit --author "you" --date "date from svn" [08:04:32] in line 583 [08:04:42] ohh funny [08:04:49] so that is just a whitespace apparently [08:04:56] as you said, the change was already in Git [08:05:00] so I did a mistake there :/ [08:05:09] yea, so i thought you are just fixing a special char or whitespace i put in there by accident [08:05:14] I did have some conflict when doing the git stuff [08:05:19] did rebase / fixed conflict [08:05:30] ends up the conflict was just because of that whitespace :/ [08:05:35] heh [08:05:49] arr, well, thanks for cleaning it up [08:06:03] we definitely need to clean that up [08:06:06] and use only git [08:06:11] should we merge it? [08:06:22] 15161 [08:06:24] or not [08:08:26] that is on production already [08:08:38] eh, yeah, and the dependencies... [08:08:40] let me check on fenari [08:08:44] it has deps [08:08:47] both ways [08:08:51] alright [08:09:52] hmm [08:10:00] * 8d6536c - add labs->labsconsole redirect per gerrt 14509 / RT-2402 (13 hours ago) [08:10:04] that is the local change [08:10:27] ahh that was already there * 4865b73 - bug 36884 - redirect labs.wm.o -> labsconsole (5 days ago) [08:10:29] greeat [08:10:32] so just abandon it [08:10:35] I will rebase the other change [08:10:42] aka abandon https://gerrit.wikimedia.org/r/#/c/15161/ [08:11:19] New review: Hashar; "Already merged with I977f90acf35f139d3cb8dc9036fe688f9210978b" [operations/apache-config] (master) C: -1; - https://gerrit.wikimedia.org/r/15161 [08:11:31] Change abandoned: Hashar; "Already there as I977f90acf35f139d3cb8dc9036fe688f9210978b" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/15161 [08:11:36] ah, thanks [08:12:35] New patchset: Hashar; "favicon.php test" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/15162 [08:12:57] New review: Hashar; "Rebased to remove the faulty dependency on I7c386f80" [operations/apache-config] (master) C: 0; - https://gerrit.wikimedia.org/r/15162 [08:13:04] https://gerrit.wikimedia.org/r/#/c/15162/ can be merged now I guess [08:15:45] does it also mean you want to deploy it right away after git merge? [08:16:43] eh, or commit to svn .. [08:17:27] http://en.wikipedia.org/w/favicon.php [08:17:37] $wgFavicon is configured incorrectly, it must be set to something other than /favicon.ico <-- expected? [08:18:11] mutante: that is Tim stuff I think [08:18:18] ohhh [08:18:37] anyway I think the favicon is http://en.wikipedia.org/favicon.ico anyway [08:18:50] Tim is rebuilding our favicon stuff to be served from the PHP file [08:18:58] that would let us have the same docroot for all virtual hosts [08:18:59] but the rewrite rule in that gerrit change ends in /w/favicon.php [L] [08:19:07] versus the current lame situation where we have a docroot per domain [08:19:16] oh [08:19:23] maybe the favicon.php script look at the request? [08:19:27] and you can't call it directly [08:19:35] I have no idea honestly, would need to look at the code [08:19:42] yea, and then says $wgFavicon is configured incorrectly [08:19:44] anyway, that is Tim handling it so it is safe :-] [08:19:51] so i guess it needs a change to mediawiki config too [08:20:06] dependency between a change in mediawiki-config and one in apache-config :p [08:20:41] ok, maybe it just does it because i call it directly, gotcha [08:21:14] require( "/apache/common/live-1.5/MWVersion.php" ); [08:21:14] require( getMediaWiki( "includes/WebStart.php" ) ); [08:21:16] ahhh [08:21:20] Tim is soooo smart [08:21:30] that get you a MediaWiki context :-] [08:23:03] oh,ok, so you still want the Tim review or have it merged now. it just rewrites test-favicon.ico, not favicon.ico, so i dont care [08:23:12] but you have him in as reviewer anyways [08:23:27] that is automatically set since he is the reviewer [08:23:31] but the change come from fenari [08:23:36] so it is already deployed :-] [08:23:43] sorry should have been clearer [08:23:57] I asked Ryan to merge both changes, looks like he switched to something else :-] [08:24:27] 'xterm-256color': unknown terminal type. [08:24:29] seriously [08:24:39] New review: Dzahn; "already in production per hashar. see IRC" [operations/apache-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/15162 [08:24:40] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/15162 [08:25:22] updating git copy on fenari [08:25:48] FUCKING GROOUP WRITE [08:25:50] sorry [08:25:53] had to swear [08:26:20] :) ok, hashar, here's a German phrase of the day: http://en.wiktionary.org/wiki/was_zum_Kuckuck [08:26:40] Kuckuck ahaha [08:26:41] what the cuckoo [08:26:43] sounds funny [08:27:23] could you update group write on fenari please? chmod -R g+w /home/wikipedia/conf/httpd [08:27:32] seems some people have an incorrect uname [08:28:07] /home/wikipedia/conf/httpd/.git/objects/ has some dir which are not group writable :( [08:29:05] eh, wait, that would let every wikidev member commit, is that intended [08:29:25] uhm,,looking at .git [08:29:32] I guess that is the point ;) [08:30:01] the svn repos is wikidev writable too [08:30:41] (ok ideally we would have a unique code for each application, create a group for each application and only add in that group people allowed to handle it ) [08:30:50] but then we will have a hugely complicated system ;-] [08:32:52] hashar: try again [08:33:19] looks great [08:33:20] thx [08:33:20] !log chmod -R g+w /home/wikipedia/conf/httpd on fenari to fix group write on .git/objects [08:33:28] Logged the message, Master [08:33:34] so hmm [08:34:49] yes, it looked pretty mixed up in .git/objects, some dirs with +w and some without [08:35:45] I guess some people don't have umask 0002 [08:36:12] !log synced /h/w/conf/httpd git and svn repositories [08:36:20] Logged the message, Master [08:40:23] hashar: so the process would be to go there, git pull, svn commit, sync-apache now? [08:44:40] I did an email on ops [08:44:40] on June 27th [08:44:40] so you could either use Gerrit or local hack [08:44:40] on Gerrit that would be what you say: submit change, git pull on fenari, svn commit locally, sync-apache [08:44:41] on local hacking : git remote update, git commit, git push master:refs/for/master, submit in gerrit, syn commit locally, sync-apache [08:44:41] feel free to mail ops [08:44:41] or I will again [08:44:41] cool, tyvm ,just confirming again after the sync [08:44:41] i would not go the local hacking way then [08:44:41] # shutdown -h now [08:44:41] bash: /sbin/shutdown: cannot execute binary file [08:44:41] having fun with labs :-]]]]]]]] [08:45:06] EYOUDONTEVENDAREDOTHIS [08:45:17] saper: aren't you at WikiMania? [08:45:19] i just could not do the "git pull" part of it until the sync now, so all cool [08:45:22] no [08:47:29] hashar: echo 1 > /proc/sys/kernel/sysrq ; echo o > /proc/sysrq-trigger ? [08:47:44] * hashar knows nothing about /proc/ [08:47:48] I just deleted the instance hehe [08:49:00] you remind me, i need to check instances too [08:50:40] http://wikitech.wikimedia.org/view/Apache :p [08:51:09] can i paste your lines above there re: config deployment [08:53:32] mutante: better to copy paste from mail in ops I think [08:54:00] that page also need to be updated with Jeff_Green Apache test script :-) [08:54:17] I wish we Apache had an unit testing suite to verify our conf [08:54:18] OR [08:54:30] we could migrate to lighthttpd [08:55:56] ack, i did use Jeff's script by the way on the recent changes [08:56:39] Tim did not know about it [08:56:47] so might be good to mail everyone in ops about it [08:57:37] Jeff_Green: ^ [09:04:15] hashar: http://wikitech.wikimedia.org/index.php?title=Apaches&action=historysubmit&diff=49328&oldid=48498 [09:04:35] New patchset: Hashar; "new role::jobrunner" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15434 [09:04:50] mutante: well done :) [09:05:09] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15434 [09:13:21] Change abandoned: Hashar; "There is already role::applicationserver::job runner" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15434 [09:17:14] creating warning template on wikitech [09:32:05] PROBLEM - Puppet freshness on nfs2 is CRITICAL: Puppet has not run in the last 10 hours [09:33:48] hashar: i can't find the mail, June 27th is when we did the jenkins upgrade [09:34:11] hashar: just fyi, maybe it didnt arrive [09:35:59] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [09:37:27] mutante: forwarded it to you [09:37:54] thx [09:50:12] New patchset: Hashar; "Precise: get rid of php5-fpm" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15435 [09:50:34] mutante: ^^^ that is Precise installing php5-fpm. It seems to be a fast CGI daemon serving php which is unneeded since we use Apache. [09:50:46] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15435 [10:00:43] New patchset: Hashar; "varnish config for bits.beta.wmflabs.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13304 [10:01:15] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13304 [10:01:57] New review: Hashar; "Patchset 3 update the apaches IP on beta." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/13304 [10:02:42] New patchset: Hashar; "varnish config for bits.beta.wmflabs.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13304 [10:03:16] New review: Hashar; "Patchset 4 is a rebase on latest commit of the 'production' branch." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/13304 [10:03:16] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13304 [10:06:45] mutante: would you mind merging https://gerrit.wikimedia.org/r/#/c/13304/ ? :( [10:07:00] it is to let me install a varnish instance on beta labs [10:07:19] I have copied the bits.wm.org Varnish configuration, removed some geoip stuff from it [10:07:28] then just call varnish::instance {} [10:07:55] the list of apaches is hardcoded in the role::cache::bits::labs class cause our Nova configuration interface does not seem to let us pass an array to a variable :/ [10:10:19] hashar, "It's official. PHP-FPM is no longer marked as "experimental" as of PHP 5.4.0RC2." [10:10:37] hashar: i was reading http://php-fpm.org/ because of the other change [10:10:52] seems to be yet another web server [10:10:57] I did not really looked at it :-] [10:11:03] FastCGI implementation with some additional features useful for sites of any size, especially busier sites. [10:11:12] we are a busier site [10:11:31] but i dont know if anybody wanted it, as you say it is just pulled in as dependency by php5 [10:11:43] ohh [10:11:43] http://php-fpm.org/wiki/Features#request_slowlog_timeout [10:11:48] yea [10:11:54] gives you a stack trace whenever a script is slow [10:13:47] yea, so i was like true, we dont use it, but maybe somebody even wants to look at it [10:13:54] shrug [10:15:38] New review: Dzahn; "true, it doesn't exist on lucid but the features it has don't even sound so bad? "for busier sites",..." [operations/puppet] (production); V: 1 C: 1; - https://gerrit.wikimedia.org/r/15435 [10:16:53] mutante: as I said, we don't use it :-] [10:17:07] so no point of having php5-fpm listening on our servers [10:19:41] ok, and you could remove it without removing more php packages? [10:19:58] indeed [10:20:14] The following packages will be REMOVED: [10:20:15] php5-fpm [10:20:20] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15435 [10:21:03] hashar: now let me sneak in another apache config one: [10:21:17] https://gerrit.wikimedia.org/r/#/c/15237/ what shall we do with this [10:21:47] ohh [10:21:50] did you see any wikipedie.cz changes? [10:22:03] that is a discussion jeremy and Ryan had yesterday [10:22:24] apparently the previous change ( https://gerrit.wikimedia.org/r/#q,I6f16f6498fe3bb,n,z ) had an issue [10:22:31] about the technical part or about what they want to achieve with the redirect [10:22:39] hmm [10:23:02] IIRC, http://wikipedie.cz/experti_na_prirodu needs to redirect on some cs.wikipedia.org page [10:23:03] let's try to clean up all in apache-config repo [10:23:26] but there is a rewrite rule that just redirect it to http://cs.wikipedia.org/experti-na_prirodu [10:23:37] so change 15237 moves the rewrite rule before the rule that conflict [10:25:31] mutante: I am amending the commit message [10:25:33] New patchset: Hashar; "wikipedie.cz/experti_na_prirodu followup" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/15237 [10:25:57] New review: Hashar; "Patchset 2 add an explanation in the commit message." [operations/apache-config] (master) C: 0; - https://gerrit.wikimedia.org/r/15237 [10:26:41] New review: Hashar; "The rule for http://wikipedie.cz/experti_na_prirodu need to be placed" [operations/apache-config] (master) C: 1; - https://gerrit.wikimedia.org/r/15237 [10:26:50] mutante: https://gerrit.wikimedia.org/r/#/c/15237/ looks good to me [10:27:00] just an issue with the RewriteRule order in redirects.conf [10:28:11] thanks hashar, sorry phone, brb [10:28:14] okkk [10:32:55] PROBLEM - Puppet freshness on db1029 is CRITICAL: Puppet has not run in the last 10 hours [10:35:41] New review: Dzahn; "yep yep, just the same thing in another place to hit this rule first" [operations/apache-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/15237 [10:35:42] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/15237 [10:58:36] New patchset: Hashar; "ignore /archive & /broken" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/15437 [10:58:37] New patchset: Hashar; "rm httpd.conf now in puppet" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/15438 [10:58:37] New patchset: Hashar; "conf for secure.wikimedia.org" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/15439 [10:58:38] New patchset: Hashar; "sync wrapper to sync-apache" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/15440 [11:13:18] !log /home/wikipedia/conf/httpd cleaned out the svn repository. That directory is now 100% under git tracking yeah!!! Thanks TIm! [11:13:26] Logged the message, Master [11:16:24] PROBLEM - Apache HTTP on srv266 is CRITICAL: Connection refused [11:16:36] !log yay @ svn removal [11:16:43] Logged the message, Master [11:16:45] just wondering [11:16:49] why is this not just in puppet? [11:18:33] afaik because people wanted a way to change it on all servers at once and not wait for puppet runs on all [11:18:52] but there was some talk about it at office as well [11:20:53] Faidon told me we could potentially run puppet and ask it to just run one class [11:21:05] something like: puppetd --apply apache::config [11:21:20] so we could dsh such an hypothetical command to all apaches :) [11:21:26] that might kill puppetmaster though [11:22:05] and good morning mark :) [11:23:31] mark: will you be there this afternoon for some varnish change review ? Had to hack something for labs https://gerrit.wikimedia.org/r/#/c/13304/ [11:23:37] though I am going lunch right now [11:25:24] RECOVERY - Apache HTTP on srv266 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.445 second response time [11:26:16] also getting some food. bbl [11:26:23] no i'll be gone by end of afternoon [11:26:42] mark: well if you got sometime to review my change that would be great [11:26:43] * mark doesn't understand gerrit's access rules [11:27:05] would ask demon [11:30:38] ah [11:30:41] it's just its stupid UI [11:31:58] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/14985 [11:31:59] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/14983 [11:31:59] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/14982 [11:32:01] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/14981 [11:32:02] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/14980 [11:32:03] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/14979 [11:32:03] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/14978 [11:32:04] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/14977 [11:32:05] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/14976 [11:32:06] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/14975 [11:32:06] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/14974 [11:32:07] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/14973 [11:32:08] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/14972 [11:33:33] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15146 [11:33:34] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15145 [11:33:35] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15144 [11:33:36] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15143 [11:33:36] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15142 [11:33:37] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15141 [11:33:38] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15140 [11:33:38] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15139 [11:33:39] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15138 [11:33:40] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15137 [11:33:40] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15136 [11:33:41] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15135 [11:33:42] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15134 [11:33:43] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15133 [11:33:43] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15132 [11:33:44] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15131 [11:33:45] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15130 [11:33:45] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15129 [11:33:46] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15128 [11:33:47] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15127 [11:33:48] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15126 [11:33:48] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15125 [11:33:49] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15124 [11:33:50] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15121 [11:33:50] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15120 [11:33:51] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15119 [11:33:52] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15118 [11:33:52] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15117 [11:33:53] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15116 [11:33:54] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15115 [11:33:54] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15114 [11:33:57] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15113 [11:33:59] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15112 [11:34:00] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15111 [11:34:00] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15110 [11:34:01] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15109 [11:34:02] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15108 [11:34:03] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15107 [11:34:03] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15106 [11:34:04] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15105 [11:34:05] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15104 [11:34:05] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15103 [11:34:06] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15102 [11:34:07] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15101 [11:34:07] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15100 [11:34:08] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15099 [11:34:09] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15098 [11:34:09] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15097 [11:34:10] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15096 [11:34:11] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15095 [11:34:12] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15094 [11:34:12] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15093 [11:34:13] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15092 [11:34:14] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15091 [11:34:14] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15090 [11:34:15] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15089 [11:34:16] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15088 [11:34:16] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15087 [11:34:17] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15086 [11:34:18] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15085 [11:34:18] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15084 [11:34:19] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15083 [11:34:20] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15082 [11:34:21] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15081 [11:34:21] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15080 [11:34:22] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15079 [11:34:23] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15078 [11:34:23] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15077 [11:34:24] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15076 [11:34:25] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15075 [11:34:25] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15074 [11:34:26] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15073 [11:34:27] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15072 [11:34:27] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15071 [11:34:28] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15070 [11:34:29] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15069 [11:34:30] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15068 [11:34:30] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15067 [11:34:31] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15066 [11:34:32] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15065 [11:34:32] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15064 [11:34:33] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15063 [11:34:34] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15062 [11:34:35] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15061 [11:34:35] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15060 [11:34:36] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15059 [11:34:36] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15058 [11:34:37] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15057 [11:34:38] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15056 [11:34:39] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15055 [11:34:39] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15054 [11:34:40] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15053 [11:34:41] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15052 [11:34:41] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15051 [11:34:42] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15050 [11:34:43] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15049 [11:34:43] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15048 [11:34:44] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15047 [11:34:45] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15046 [11:34:46] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15045 [11:34:46] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15044 [11:34:47] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15043 [11:34:48] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15042 [11:34:48] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15041 [11:34:49] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15040 [11:34:50] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15039 [11:34:51] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15038 [11:34:51] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15037 [11:34:52] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15036 [11:34:52] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15035 [11:34:53] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15034 [11:34:54] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15033 [11:34:55] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15032 [11:34:55] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15031 [11:34:56] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15030 [11:34:57] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15029 [11:34:57] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15028 [11:34:58] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15027 [11:34:59] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15026 [11:34:59] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15025 [11:35:00] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15024 [11:35:01] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15023 [11:35:01] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15022 [11:35:03] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15021 [11:35:05] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15020 [11:35:06] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15019 [11:35:06] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15018 [11:35:07] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15017 [11:35:08] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15016 [11:35:09] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15015 [11:35:09] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15014 [11:35:10] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15013 [11:35:11] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15012 [11:35:11] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15011 [11:35:12] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15010 [11:35:13] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15009 [11:35:13] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15008 [11:35:14] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15007 [11:35:15] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15006 [11:35:15] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15005 [11:35:16] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15004 [11:35:17] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15003 [11:35:18] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15002 [11:35:18] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15001 [11:35:19] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15000 [11:35:20] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/14999 [11:35:20] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/14998 [11:35:21] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/14997 [11:35:22] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/14996 [11:35:22] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/14995 [11:35:23] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/14994 [11:42:32] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15149 [11:42:32] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15148 [11:42:33] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15147 [11:42:34] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15123 [11:42:35] Change merged: Mark Bergsma; [operations/debs/varnish] (testing/persistent) - https://gerrit.wikimedia.org/r/15122 [11:57:00] backkk [12:07:31] PROBLEM - Host srv206 is DOWN: PING CRITICAL - Packet loss = 100% [12:21:12] New review: Hashar; "Deployed on 'beta' instance deployment-cache-bits02 using puppetmaster::self. It is happily serving ..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/13304 [12:33:34] New patchset: Hashar; "docroot for deployment.wikimedia.beta.wmflabs.org" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/14099 [12:34:09] New review: Hashar; "I have removed the faulty PHP" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/14099 [12:34:45] New patchset: Hashar; "docroot for deployment.wikimedia.beta.wmflabs.org" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/14099 [12:35:07] Change merged: Hashar; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/14099 [12:37:40] PROBLEM - Puppet freshness on mw24 is CRITICAL: Puppet has not run in the last 10 hours [12:47:43] PROBLEM - Puppet freshness on mw1102 is CRITICAL: Puppet has not run in the last 10 hours [13:02:21] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [13:16:12] New review: Mark Bergsma; "Wouldn't it be better to parameterize the production bits stuff and use it for labs? If we diverge l..." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/13304 [13:16:20] hashar: ^ review [13:28:52] New review: Mark Bergsma; "You should use a git-buildpackage repo layout (separate upstream/debian branches), see [[wikitech::G..." [operations/debs/ircd-ratbox] (master); V: 0 C: -1; - https://gerrit.wikimedia.org/r/12425 [13:44:00] New review: Hashar; "I tried reusing the templates/varnish/bits.inc.vcl.erb template but when enable_geoiplookup is false..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/13304 [13:44:08] mark: thanks mark [13:44:29] now I remember why I copy pasted the varnish conf :-D The template renders an invalid conf whenever geoip_lookup is absent/false [13:46:58] so fix that [13:47:49] finding out if ERB supports else :-D [13:48:47] <% else %> should work? [13:50:18] apparently [13:56:51] grmblblbl [13:58:49] mark: any hint about how to test an erb template? Puppet doc says `erb -P -x -T '-' mytemplate.erb | ruby -c` but I would like to set a puppet variable (aka enable_geoiplookup) [13:59:25] just test in labs [14:01:35] New patchset: Hashar; "Fix bits VLC when enable_geoiplookup is disabled" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15445 [14:02:09] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15445 [14:02:21] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [14:14:11] New patchset: Ottomata; "filters.oxygen.erb - Saudi Telecom IP update to match https://office.wikimedia.org/wiki/Partner_IP_Ranges#Saudi_Telecom_.28ST.29" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15446 [14:14:48] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15446 [14:16:29] notice: /Stage[main]/Role::Cache::Bits::Labs/Varnish::Instance[bits-labs]/Service[varnish]/ensure: ensure changed 'stopped' to 'running' [14:16:30] yeahhh [14:18:52] New patchset: Hashar; "Fix bits VLC when enable_geoiplookup is disabled" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15445 [14:19:25] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15445 [14:19:37] New review: Hashar; "Patchset2" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/15445 [14:26:11] New patchset: Hashar; "Fix bits VLC when enable_geoiplookup is disabled" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15445 [14:26:44] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15445 [14:30:51] Logged the message, Master [14:37:17] New patchset: Hashar; "varnish config for bits.beta.wmflabs.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13304 [14:37:50] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13304 [14:39:19] New review: Hashar; "Patchset 4 rebase on top of I033e5b7f: Fix bits VLC when enable_geoiplookup is disabled" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/13304 [14:41:09] testing again [14:41:18] I love labs and tests :D [14:41:34] although it takes a good hour to do any minor change / test them [14:41:55] but in the end, when the change is applied we can forget about it!! [14:43:34] RECOVERY - Host search32 is UP: PING OK - Packet loss = 0%, RTA = 0.22 ms [14:47:07] New patchset: Hashar; "varnish config for bits.beta.wmflabs.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/13304 [14:47:42] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/13304 [14:50:21] mark: the VCL fix for bits when geoiplookup is disabled is at https://gerrit.wikimedia.org/r/#/c/15445/3 [14:51:22] New review: Hashar; "Patchset 6 had an incorrect realm name: "wmflabs" -> "labs"." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/13304 [14:51:42] yeahhh http://bits.beta.wmflabs.org/static-master/skins/vector/images/search-ltr.png?303-4 [14:52:41] New review: Hashar; "Patchset 6 deployed on labs:" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/13304 [14:53:11] mark: and the bits.beta.wmflabs.org now use the same VCL from production as of https://gerrit.wikimedia.org/r/#/c/13304/6 [14:53:26] just had to varnish::instance{ vcl => 'bits } ;-] [15:03:20] New patchset: Ottomata; "Giving access to Haitham Shammaa on stat1 for wikipedia arabic fundraiser analysis." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15448 [15:03:54] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15448 [15:07:36] New patchset: Ottomata; "statistics.pp - rsyncing arabic-banner logs to stat1 for Haitham Shammaa." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15449 [15:08:09] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15449 [15:11:06] New patchset: Ottomata; "Giving access to Haitham Shammaa on stat1 for wikipedia arabic fundraiser analysis." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15448 [15:11:39] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15448 [15:14:22] PROBLEM - Puppet freshness on ms3 is CRITICAL: Puppet has not run in the last 10 hours [15:19:19] PROBLEM - Host srv278 is DOWN: PING CRITICAL - Packet loss = 100% [15:19:55] RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms [15:20:25] hiya all [15:20:32] anyone around to help me merge a few really easy puppet commits? [15:20:34] mutante maybe? [15:23:49] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [15:25:19] PROBLEM - Puppet freshness on ms2 is CRITICAL: Puppet has not run in the last 10 hours [15:50:22] PROBLEM - Puppet freshness on ms1 is CRITICAL: Puppet has not run in the last 10 hours [15:50:49] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.057 second response time [15:51:43] RECOVERY - Host sq36 is UP: PING OK - Packet loss = 0%, RTA = 0.64 ms [15:54:52] PROBLEM - Backend Squid HTTP on sq36 is CRITICAL: Connection refused [15:55:19] PROBLEM - Frontend Squid HTTP on sq36 is CRITICAL: Connection refused [16:03:52] RECOVERY - Backend Squid HTTP on sq36 is OK: HTTP OK HTTP/1.0 200 OK - 27399 bytes in 0.019 seconds [16:04:10] RECOVERY - Frontend Squid HTTP on sq36 is OK: HTTP OK HTTP/1.0 200 OK - 27546 bytes in 0.008 seconds [16:13:47] PROBLEM - NTP on sq36 is CRITICAL: NTP CRITICAL: Offset unknown [16:39:52] PROBLEM - MySQL Replication Heartbeat on db12 is CRITICAL: CRIT replication delay 214 seconds [16:39:52] PROBLEM - MySQL Replication Heartbeat on db42 is CRITICAL: CRIT replication delay 214 seconds [16:40:10] PROBLEM - MySQL Slave Delay on db12 is CRITICAL: CRIT replication delay 233 seconds [16:40:19] PROBLEM - MySQL Replication Heartbeat on db1042 is CRITICAL: CRIT replication delay 241 seconds [16:40:19] PROBLEM - MySQL Replication Heartbeat on db1001 is CRITICAL: CRIT replication delay 242 seconds [16:40:55] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 277 seconds [16:41:04] PROBLEM - MySQL Replication Heartbeat on db1043 is CRITICAL: CRIT replication delay 286 seconds [16:41:04] PROBLEM - MySQL Replication Heartbeat on db1033 is CRITICAL: CRIT replication delay 287 seconds [16:42:34] RECOVERY - MySQL Replication Heartbeat on db1043 is OK: OK replication delay 0 seconds [16:42:43] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 231 seconds [16:42:52] RECOVERY - MySQL Replication Heartbeat on db12 is OK: OK replication delay 1 seconds [16:43:10] RECOVERY - MySQL Slave Delay on db12 is OK: OK replication delay 1 seconds [16:43:19] RECOVERY - MySQL Replication Heartbeat on db1042 is OK: OK replication delay 0 seconds [16:43:19] RECOVERY - MySQL Replication Heartbeat on db1001 is OK: OK replication delay 0 seconds [16:43:46] PROBLEM - MySQL Slave Delay on db42 is CRITICAL: CRIT replication delay 293 seconds [16:43:55] RECOVERY - MySQL Replication Heartbeat on db1047 is OK: OK replication delay 8 seconds [16:44:04] RECOVERY - MySQL Replication Heartbeat on db1033 is OK: OK replication delay 0 seconds [16:44:13] RECOVERY - MySQL Slave Delay on db1047 is OK: OK replication delay 1 seconds [16:47:31] RECOVERY - MySQL Replication Heartbeat on db42 is OK: OK replication delay 16 seconds [16:48:16] RECOVERY - MySQL Slave Delay on db42 is OK: OK replication delay 0 seconds [17:00:52] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [17:44:06] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [17:57:51] WTF? There is a misc::secure but it's NOT USED on singer? [18:19:27] New patchset: Catrope; "Actually use misc::secure to run secure.wikimedia.org on singer" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15458 [18:20:01] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15458 [18:24:34] New review: Reedy; "Can't we just shoot singer?" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/15458 [18:25:07] Heya LeslieCarr [18:25:20] you got a bit of time this afternoon to help me finish merging that base monitoring stuff? [18:25:25] (and maybe a one or a few more?) [18:25:33] oh she's not here [18:52:13] Heya LeslieCarr [18:52:13] you got a bit of time this afternoon to help me finish merging that base monitoring stuff? [18:52:13] (and maybe a one or a few more) [18:54:03] ottomata: yay [18:54:08] let's do it now [18:54:49] yay [18:54:50] yes [18:55:04] https://gerrit.wikimedia.org/r/#/c/14911/ [18:55:19] i'm not sure if I completely understood your last comment [18:55:29] but I did need to c hange something [18:55:48] but I assume by base::monitor::host and base::monitoring::host should be here [18:55:54] you just wanted me to include it below all the other includes [18:56:50] yeah [18:57:04] ok cool, then that's done [18:57:14] also my class name there was wrong, and I fixed that [18:57:22] it is base::monitoring::host, not base::monitor::host [19:00:25] New patchset: Pyoungmeister; "cleanup of apache.pp, mediawiki, roles/apache, etc" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15459 [19:01:01] Change abandoned: Pyoungmeister; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15242 [19:01:01] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15459 [19:02:35] checking now... [19:02:48] cmjohnson1: you mean search [19:02:48] ok [19:02:50] :) [19:02:52] sounds good! [19:03:43] Logged the message, Master [19:04:01] one spelling error and then we're good to go :) [19:04:47] ah k [19:06:18] New patchset: Ottomata; "base.pp,site.pp - parameterizing contact_group for base::monitoring::host" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/14911 [19:06:54] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/14911 [19:07:09] ok, easy done LeslieCarr [19:07:52] ottomata: on the puppet change you wanted me to review, i think you meant to have role::logging on each host, not misc::logging [19:08:06] er, misc::udp2log [19:08:31] yay!!! [19:09:24] oh, nvm [19:09:27] herp derp [19:09:47] ready to merge it ottomata? [19:10:38] yup, ready, now, I only ahve access to some of these machines [19:10:41] actually [19:10:47] um, i don't think I have access to any of the relevant ones [19:10:55] since these are mainly virtual resources [19:10:56] right? [19:10:59] so jaja, i'm ready [19:11:03] but I won't be able to check it [19:11:18] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15459 [19:11:41] and we won't really know if it works until something goes bad [19:11:46] wonder if there is a way we can simulate [19:11:46] hm [19:11:54] anyway, yes LeslieCarr, I'm ready if you are [19:12:23] !log finally stopped using my production ssh key on labs. [19:12:30] Logged the message, Master [19:13:56] yay [19:14:06] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/14911 [19:14:34] okey dokey [19:15:01] merging now... [19:15:05] ok cool [19:15:09] I'm not sure where these things go into nagios [19:15:21] but I guess the best check we can do is to make sure the analytics contact group now appears wherever it should [19:15:28] let's try a puppet pull… [19:15:30] i guess on the nagios server? dunno [19:16:02] yeah, the checks happen the nagios server pulls all the possible checks [19:16:15] going to pull on spence now [19:16:25] ok, so there should be a config file that changes the contact group for some of the monitor hosts then [19:16:29] check emery or something [19:16:46] ( i guess anyway, i have no idea :) ) [19:16:53] ahha [19:17:13] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: System_role[misc::log-collector] is already defined in file /var/lib/git/operations/puppet/manifests/role/logging.pp at line 8; cannot redefine at /var/lib/git/operations/puppet/manifests/role/logging.pp:29 on node emery.wikimedia.org [19:17:45] cmjohnson1: thanks! [19:17:51] ah yeah, i'll remove it from line 8 [19:18:12] New patchset: Lcarr; "removing dupe definition" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15462 [19:18:46] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15462 [19:19:43] ah oops, thanks [19:20:43] yay working [19:20:57] any reason to also not do a dist-upgrade ? [19:22:33] eh? [19:22:37] are you asking me? [19:23:15] yeah [19:23:21] oh on the logging machines [19:23:26] does that upgrade the OS? [19:23:44] doesn't upgrade it to precise or anything ;) just upgrades the packages to the latest minor version [19:23:50] um, i unno? [19:23:55] i know of no reason not to do that [19:25:50] if you had manually changed a package or something ... [19:25:59] though you shouldn't have ddone that ;) [19:26:04] so woot [19:32:02] ok [19:32:48] PROBLEM - Puppet freshness on nfs2 is CRITICAL: Puppet has not run in the last 10 hours [19:32:57] New review: Catrope; "I did this so that I could put in redirect rules from secure.wikimedia.org/$wiki/$lang to $lang.$wik..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/15458 [19:33:03] !log shutting down srv203 for hardware checking per rt3110 [19:33:11] Logged the message, RobH [19:33:18] cmjohnson1: ^ all yours [19:33:34] Logged the message, Master [19:33:39] quite welcome [19:34:06] anyone know the message size limit for udp2log messages? [19:35:09] New patchset: Catrope; "Actually monitor the survey.wm.o cert on argon" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15463 [19:35:30] PROBLEM - Host srv203 is DOWN: PING CRITICAL - Packet loss = 100% [19:35:43] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15463 [19:36:51] PROBLEM - Puppet freshness on nfs1 is CRITICAL: Puppet has not run in the last 10 hours [19:37:41] great, thanks LeslieCarr (sorry was distracted for a moment with some C code) [19:37:43] uhhhmmm [19:37:49] no problem [19:37:52] can you verify that the contact group for emery has changed? [19:37:59] wherever it is supposed to have been changed [19:44:31] New patchset: Lcarr; "removing mailq.cfg from icinga" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15465 [19:44:49] whenever spence finally gets puppet running.... [19:45:06] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15465 [19:46:02] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15465 [19:46:31] ha, ok... [19:52:15] LeslieCarr, not sure what you are working on, but I have some other things that need some approval too :D [19:52:22] they shoudl all be easier than that one we just did [19:52:30] and I can verify that they all work myself [19:52:33] if they get merged [19:53:03] ottomata: ok [19:53:48] we'll start with the larger one (still pretty easy!) to make diederik happy [19:53:49] https://gerrit.wikimedia.org/r/#/c/14922/ [19:53:50] yay, contact groups are changed [19:53:57] yayayayayayayyayayay [19:53:58] thank you! [19:55:19] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/14922 [19:56:44] danke danke! [19:57:36] next up [19:57:37] https://gerrit.wikimedia.org/r/#/c/15446/ [19:57:47] oh uh [19:57:49] ok wait [19:57:50] not that one [19:57:52] gotta fix one thing :p [19:58:02] how about this: [19:58:03] https://gerrit.wikimedia.org/r/#/c/15448/ [19:59:30] New patchset: Ottomata; "filters.oxygen.erb - Saudi Telecom IP update to match https://office.wikimedia.org/wiki/Partner_IP_Ranges#Saudi_Telecom_.28ST.29" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15446 [20:00:10] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15446 [20:05:39] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15446 [20:08:09] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15448 [20:08:58] woohoo [20:09:18] one more for today [20:09:21] https://gerrit.wikimedia.org/r/#/c/15449/ [20:10:33] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15449 [20:10:45] merge mania!!! [20:10:54] merged on puppetmaster [20:11:22] yeeeweeoooo [20:11:30] thanks so much, ok going to run puppet in relevant places [20:11:33] hashar: I'm goig to revert your jobrunner role commit. I already made something in role/apache.pp. does that work for you? [20:11:52] notpeter: which commit / change ? [20:12:08] I got several :-D https://gerrit.wikimedia.org/r/#/dashboard/24 [20:12:23] https://gerrit.wikimedia.org/r/#/c/15434/ [20:12:43] is the smae as role::applicationserver::jobrunner [20:12:50] I have abandoned that one [20:12:52] except that it moves away from the cruft in site.pp [20:12:53] kk [20:13:21] the reason was : There is already role::applicationserver::job runner [20:13:21] hehe [20:13:25] :) [20:13:27] excellent [20:13:36] RECOVERY - Host srv203 is UP: PING OK - Packet loss = 0%, RTA = 0.43 ms [20:14:11] the manifest/role/apache.pp could use some improvement [20:14:44] the role::applicationserver::labs seems to reimplement some part of the production [20:14:46] hashar: what would you like changed? [20:14:54] oh, yeah [20:15:03] I thought about merging them into one class [20:15:06] for example it does not support the apache => false [20:15:14] so we end up having job runner running apache :-] [20:15:16] not a big issue though [20:15:21] ah, gotcha [20:15:34] want me to make all the same options available for the labs roles? [20:15:56] looks like the main differences are: apaches::monitoring::labs, nfs::apache::labs, [20:16:04] and the accounts [20:16:12] and labs does not have admins:: and accounts::l10nupdate [20:16:13] yeah [20:16:13] :) [20:16:14] ahhh LeslieCarr, a problem with the haithams change [20:16:18] got a commit coming at you in a sec [20:16:18] not sure how that could be solved [20:16:31] and the geoip classes [20:16:31] ok [20:16:37] maybe role::aplicationserver::common::prod and role::aplicationserver::common::labs [20:16:46] oh, yeah, that's legit [20:16:48] I can do that [20:16:55] then have common include one of them based on $::realm [20:17:04] yep! sounds good [20:17:06] New patchset: Asher; "make blondel writeable" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15499 [20:17:10] oh hmm no [20:17:12] PROBLEM - Apache HTTP on srv203 is CRITICAL: Connection refused [20:17:17] I think Faidon and Mark don't like the if( $::realm ) [20:17:27] so maybe add a new parameter to the common class such as : labs => false [20:17:36] yeah, iunno. the flexibility of two seperate classes is kinda nice [20:17:40] and we can have the labs roles to just labs => true [20:17:40] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15499 [20:17:46] New patchset: Ottomata; "site.pp - accounts::haithams, not admins::haithams" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15500 [20:17:52] oh, sure [20:17:55] that works too [20:17:57] cool! [20:18:14] as you said, we will probably want to keep the role::applicationserver::labs to sneak in some exceptions local to labs [20:18:17] ok LeslieCarr ^ [20:18:20] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15500 [20:18:26] but factoring the two commons would be great :-] [20:18:32] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15499 [20:18:35] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15500 [20:18:48] notpeter: have you heard of anything about migrating Gerrit from db9 to the new misc DB ? [20:18:56] danke! [20:19:54] New patchset: Ryan Lane; "Initial commit of the new deployment system" [operations/deployment] (master) - https://gerrit.wikimedia.org/r/8732 [20:21:13] ottomata: anything else? i was planning on heading out in a minute [20:21:29] ummmmm, actually yeah real quick [20:22:39] hashar: binasher is working on that currently. it's not moving to eqiad right now, as we don't have misc db boxes in eqiad [20:22:50] but it's going onto the new master which is on ssd soon [20:23:08] ohhh so SSD is probably going to help too [20:23:29] uhoh [20:23:33] hurry hurry :) [20:23:35] aren't you afraid of SSD expiring their write limit very fast with high writing rate server? [20:23:40] uh, maybe, maybe not [20:23:41] New patchset: Ottomata; "base.pp - emacs23, not emacs32" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15501 [20:23:46] probably not [20:23:54] there it is [20:24:00] git-review takes a while sometimes [20:24:04] cehcking on my other stuff... [20:24:19] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15501 [20:24:25] done [20:24:30] i am heading out now :) [20:24:30] hashar: high write rate? where? [20:24:31] bye [20:24:36] ooooook ok [20:24:38] thanks! [20:24:41] binasher: on a database ? I have no idea honestly [20:25:03] binasher: just some users in my coworking space which were wondering how long a SSD would last on a server [20:26:00] an ssd is not going to help the gerrit database [20:26:12] binasher: apparently each cell has a limited life experancy, something like a million write. [20:26:13] the entire gerrit db is 125mb [20:26:19] hashar: indeed [20:26:49] and then people told me how my laptop will one day have no more disk space left (literraly) [20:27:01] so I panicked and grabbed a coffee :-) [20:27:01] hashar: downloading lots of pr0n will do that! [20:27:35] I should point my browser cache to a RAM disk :-D [20:28:41] New patchset: Ottomata; "statistics.pp fixing gerrit stats cron job to push" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15502 [20:29:08] hashar: the only thing this move will do for you is the query profiling [20:29:14] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15502 [20:29:30] binasher: well the query profiling will already be a great improvement [20:29:32] if we see gerrit making lots of db queries per request, that's a good indicator that moving the db to eqiad actually will help [20:29:45] we might end up finding some obvious index to add [20:29:48] or maybe something else could be done [20:29:52] yep [20:30:09] I talked about with the Gerrit people in #gerrit, apparently the application is doing lame things [20:30:41] such as $rows = select * from table; foreach( $rows as $row ) { select some field for $row, select another field for $row } [20:30:49] so it ends up doing a massive amount of queries [20:30:58] maybe it uses an ORM ;) [20:31:08] hehe [20:31:32] I guess most developers never had to optimize that [20:34:18] PROBLEM - Puppet freshness on db1029 is CRITICAL: Puppet has not run in the last 10 hours [20:37:49] New patchset: Asher; "include tcp query sampling -> ishmael on new misc cluster" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15503 [20:38:21] RECOVERY - Apache HTTP on srv203 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.038 second response time [20:38:23] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/15503 [20:38:44] hashar: that query pattern would definitely be helped by moving the db close to the app server, assuming ~20-40ms of extra latency per individual query [20:49:10] binasher: indeed ;)à [20:49:13] oh left [20:49:23] well I am heading bed now. [20:58:04] New patchset: Hashar; "$cluster was not always defined in IS.php" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/15508 [20:59:09] New review: Hashar; "We might want to mark $wmfUdp2logDest (and others globals used there) at the top of InitialiseSettin..." [operations/mediawiki-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/15508 [21:00:27] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/15508 [21:08:21] ottomata: hey [21:09:12] hiiiii [21:09:52] i'm back if there's another review you need [21:10:21] one! [21:10:37] https://gerrit.wikimedia.org/r/#/c/15502/ [21:14:09] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/15502 [21:16:45] ottomata: completed [21:16:48] and now i sign off again :) [21:16:50] bye [21:16:54] dankkeeee [21:16:59] youddabest merger ever! [21:23:04] !log rebuilding db49 (otrs slave) [21:23:12] Logged the message, Master [21:31:02] !log rebooting db49 for kernel upgrade [21:31:10] Logged the message, Master [21:32:53] PROBLEM - Host db49 is DOWN: PING CRITICAL - Packet loss = 100% [21:34:23] RECOVERY - Host db49 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [21:36:46] !log running hotbackup of db48 -> db49 (otrs / external misc) [21:36:53] Logged the message, Master [22:01:05] PROBLEM - Host mw1053 is DOWN: PING CRITICAL - Packet loss = 100% [22:39:03] PROBLEM - Puppet freshness on mw24 is CRITICAL: Puppet has not run in the last 10 hours [22:49:06] PROBLEM - Puppet freshness on mw1102 is CRITICAL: Puppet has not run in the last 10 hours [23:03:03] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [23:07:06] PROBLEM - Host mw1016 is DOWN: PING CRITICAL - Packet loss = 100%