[00:09:08] <ori_>	 about to sync-dir
[00:09:35] <logmsgbot>	 !log ori synchronized php-1.23wmf8/extensions/WikimediaEvents  'Ibe918eb96: Update WikimediaEvents to master for Ia6f758068 (Log event when new editors reach edit milestone)'
[00:09:53] <morebots>	 Logged the message, Master
[00:10:15] <logmsgbot>	 !log ori synchronized php-1.23wmf7/extensions/WikimediaEvents  'Ibe918eb96: Update WikimediaEvents to master for Ia6f758068 (Log event when new editors reach edit milestone)'
[00:10:31] <morebots>	 Logged the message, Master
[00:11:09] <yurik>	 Reedy, do you know why fluorine still doesn't have the new bug log file? Do i need to do any extra steps?
[00:16:52] <Reedy>	 Need to get someone to touch it
[00:18:01] <mutante>	 yurik: i can touch an empty file for you if you tell me exactly what and thats all you need right now
[00:19:44] <yurik>	 mutante, they don't auto-create?   Bug58676.log
[00:20:00] <yurik>	 thx!
[00:20:27] <mutante>	 yurik: no idea, where is that on fluorine?
[00:20:41] <yurik>	   /a/mw-log
[00:21:25] <yurik>	 mutante, capitalized i think
[00:21:43] <mutante>	 aa..really i almost asked
[00:21:52] <mutante>	 and removed capitalization
[00:22:51] <mutante>	 !log create /a/mw-log/Bug58676.log for yurik, touch, chown udplog ..etc
[00:22:55] <mutante>	 yurik: done
[00:23:05] <morebots>	 Logged the message, Master
[00:23:12] <yurik>	 thx mutante !  now if only i can see some messages in it ... :)
[00:23:53] <mutante>	 no problem, actually you should hope for NO messages in a bug log,no?:) i didnt read the bug yet:)
[00:23:57] <yurik>	 do you know where all the generic stuff from debug logs should go?
[00:24:19] <mutante>	 not really, using the bug number seemed good for now if it's special
[00:24:20] <yurik>	 if they don't match specific name
[00:24:32] <mutante>	 i dont know of any schemas here
[00:24:38] <yurik>	 oki, thx!
[00:25:29] <mutante>	 Invalid message parameter  .. i see
[00:25:33] <mutante>	 well good luck with debugging then
[00:25:40] <yurik>	 heh :)
[00:26:27] <mutante>	 i see some coming in yurik
[00:26:32] <mutante>	 you got those now, right
[00:26:48] <yurik>	 mutante, no, tail is blank
[00:26:51] <mutante>	 or should i paste
[00:27:07] <yurik>	 please paste (unless private of course :))
[00:27:24] <mutante>	 i just see wiki page names and id numbers and mw host numbers
[00:27:31] <mutante>	 hold on
[00:27:37] <yurik>	 should be ok... as long as you don't see cookies ;)
[00:28:24] <yurik>	 mutante, where do you see them?
[00:28:33] <mutante>	 yurik: it should NOT be capitalized
[00:28:35] <mutante>	 thats why
[00:28:50] <mutante>	 because i happened to first not capitalize it and then moved it
[00:28:57] <yurik>	 ?? that's strange:  https://gerrit.wikimedia.org/r/#/c/102604/2/includes/Message.php
[00:28:58] <mutante>	 the first one caught those in the seconds in between
[00:29:13] <yurik>	 there it uses capitalization
[00:29:28] <yurik>	 checking tin...
[00:29:32] <mutante>	 yurik: http://paste.debian.net/71862/
[00:29:55] <mutante>	 duh, wrong bug?
[00:29:59] <yurik>	 mutante, yep :)
[00:30:01] <mutante>	 sorry
[00:30:33] <mutante>	 well then, yea, it's empty
[00:30:40] <mutante>	 and capitalized
[00:30:58] <yurik>	 heh... maybe its a randomness factor of sorts? like there is some throtling... doubtful
[00:31:31] <mutante>	 yurik: fwiw, one capitalized, 2 are not .. so shrug
[00:31:39] <mutante>	 of other bug log files like that
[00:31:53] <yurik>	 yes, i saw them, but only after the patch went live
[00:31:58] <yurik>	 would have made it noncap
[00:34:04] <mutante>	 or your bug is just not happening that often 
[00:38:50] <mutante>	 well, for completeness yurik
[00:38:56] <mutante>	   0 -rw-r--r-- 1 udp2log udp2log    0 Dec 20 00:21 Bug58676.log
[00:39:12] <mutante>	 in case somebody sees later i made a mistake or whatever
[00:40:25] <yurik>	 thanks mutante , maybe something else is missing, or it will work after some cron refresh
[00:41:25] <spagewmf>	 greg-g: OK, we're ready. It's just three files to sync to 1.23wmf8, OK to do it now?
[00:42:46] <ori_>	 spagewmf: i need to do another two sync-dirs; i need another moment, please.
[00:43:12] <spagewmf>	 ori_ np, brings back happy memories of EventLogging tweaks :)
[00:43:30] <greg-g>	 spagewmf: yeah
[00:45:05] <ori_>	 jenkins jenkins jenkins jenkins jenkins jenkins jenkins
[00:46:05] <spagewmf>	 ori_ do api 500 errors show up in the "MediaWiki errors last 2hr" graph?
[00:47:48] <logmsgbot>	 !log ori synchronized php-1.23wmf7/extensions/WikimediaEvents  'I7cad8fd35fd: Update WikimediaEvents to master'
[00:48:05] <morebots>	 Logged the message, Master
[00:48:31] <logmsgbot>	 !log ori synchronized php-1.23wmf8/extensions/WikimediaEvents  'I7cad8fd35fd: Update WikimediaEvents to master'
[00:48:38] <ori_>	 spagewmf: done, sorry for the holdup
[00:48:49] <morebots>	 Logged the message, Master
[00:49:00] <ori_>	 spagewmf: do they result in PHP exceptions getting thrown? I forget
[00:50:52] <spagewmf>	 ori_: I'm not sure either.  fluorine's api.log just has the request, not the error status.  I'll look into it later today
[00:52:47] <ori_>	 spagewmf: well, I think exceptions thrown in PHP while serving an API request get logged to the exception log; they are not special. fatals ditto. but a 500 response could be coming from varnish, too, in which case it won't be in either log.
[00:54:01] <ori_>	 for example, if the backend server is slow to process the request, varnish will eventually give up, and issue a 500 response to the user
[00:54:34] <ori_>	 but the request could still complete successfully. it just won't make it to the user.
[00:57:36] <grrrit-wm>	 (03PS1) 10Tim Landscheidt: Fix quoting error in generic_vhost.erb [operations/puppet] - 10https://gerrit.wikimedia.org/r/102862 
[00:57:44] <mutante>	 RoanKattouw_away: fyi, trying to unblock a ticket you created on Wed Dec 12 00:49:54 2012 :)
[00:57:53] <mutante>	 about the times being off on LVS
[01:00:28] <grrrit-wm>	 (03PS2) 10Tim Landscheidt: Fix quoting error in generic_vhost.erb [operations/puppet] - 10https://gerrit.wikimedia.org/r/102862 
[01:06:58] <logmsgbot>	 !log spage synchronized php-1.23wmf8/extensions/Echo/formatters/BasicFormatter.php  'Bug 58705 notification fix'
[01:07:16] <morebots>	 Logged the message, Master
[01:08:32] <logmsgbot>	 !log spage synchronized php-1.23wmf8/extensions/Echo/includes/EmailFormatter.php  'Bug 58705 notification fix pt.2'
[01:08:49] <morebots>	 Logged the message, Master
[01:10:50] <grrrit-wm>	 (03CR) 10Ori.livneh: "Reviewers: please wait for me to be around before merging, so I can verify log parsing of the updated format on vanadium." [operations/puppet] - 10https://gerrit.wikimedia.org/r/102817 (owner: 10Ori.livneh)
[01:12:32] <logmsgbot>	 !log spage synchronized php-1.23wmf8/extensions/Flow/includes/Notifications/Controller.php  'Bug 58705 notification fix pt.3'
[01:12:52] <morebots>	 Logged the message, Master
[01:14:32] <spagewmf>	 greg-g bug 58705 fixed, http://tools.wmflabs.org/gerrit-patch-uploader/ working.  All because of i18n adding an innocent $wgLang->getDir() to support RTL notification icons
[01:36:26] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC  
[01:36:26] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC  
[01:55:51] <grrrit-wm>	 (03PS8) 10Springle: role and module structure for ishmael [operations/puppet] - 10https://gerrit.wikimedia.org/r/96403 (owner: 10Dzahn)
[01:58:02] <grrrit-wm>	 (03CR) 10Springle: [C: 032] "I'll watch it." [operations/puppet] - 10https://gerrit.wikimedia.org/r/96403 (owner: 10Dzahn)
[02:01:58] <grrrit-wm>	 (03PS1) 10Springle: 'vslow' should be sufficient after https://gerrit.wikimedia.org/r/100727 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102870 
[02:03:13] <grrrit-wm>	 (03CR) 10Springle: [C: 032] 'vslow' should be sufficient after https://gerrit.wikimedia.org/r/100727 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102870 (owner: 10Springle)
[02:04:39] <logmsgbot>	 !log springle synchronized wmf-config/db-eqiad.php  'vslow query load balancing'
[02:04:56] <morebots>	 Logged the message, Master
[02:16:24] <logmsgbot>	 !log LocalisationUpdate completed (1.23wmf7) at Fri Dec 20 02:16:24 UTC 2013
[02:20:06] <icinga-wm>	 PROBLEM - puppetmaster https on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[02:25:06] <icinga-wm>	 PROBLEM - HTTP on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[02:35:55] <logmsgbot>	 !log LocalisationUpdate completed (1.23wmf8) at Fri Dec 20 02:35:55 UTC 2013
[02:37:29] <jeremyb>	 mutante: 6536 is too meta
[02:37:32] <jeremyb>	 Coren: too
[02:38:47] <Coren>	 jeremyb: The James thing?
[02:39:06] <jeremyb>	 yeah
[02:42:32] <mutante>	 yea, give me code review, not more reasons to slow it down and waiting periods pls
[02:44:57] <mutante>	 jeremyb: suggestions?
[02:47:09] <spagewmf>	 wikitech.wikimedia.org not responding... is it me?
[02:48:27] <mutante>	 spagewmf:  It's not just you! http://wikitech.wikimedia.org looks down from here. 
[02:48:33] <mutante>	 http://www.downforeveryoneorjustme.com/http://wikitech.wikimedia.org/
[02:48:47] <mutante>	 sigh.. what...
[02:49:05] <mutante>	 spagewmf: checked labs channel yet. i am about to
[02:50:41] <spagewmf>	 http://www.downforeveryoneorjustme.com/http://wikitech.wikimedia.org/ agrees.  It's pingable
[02:51:32] <mutante>	 geee..why now...
[02:52:29] <springle>	 mutante: the ishamel puppet changes broke puppet on neon. could that be related?
[02:52:44] <mutante>	 springle: i didnt know it was merged
[02:52:46] <springle>	 found a bug and about to commit the fix
[02:53:04] <mutante>	 springle: it doesnt sound very related at all
[02:53:10] <mutante>	 but with puppet you never know
[02:53:22] <springle>	 no? ok. just saying :)
[02:53:23] <mutante>	 ping others, i'm looking
[02:53:30] <mutante>	 on the actual server
[02:55:19] <mutante>	 !log restarting apache on virt0, wikitech is down
[02:55:49] <mutante>	 springle: eh, back
[02:55:53] <mutante>	 spagewmf: 
[02:55:56] <icinga-wm>	 RECOVERY - HTTP on virt0 is OK: HTTP OK: HTTP/1.1 302 Found - 457 bytes in 0.071 second response time  
[02:55:57] <mutante>	 it works again
[02:56:04] <mutante>	 i dunno, apache restart
[02:56:08] <mutante>	 but it was running before too
[02:56:47] <mutante>	 springle: thanks for that, but seems entirely unrelated
[02:56:55] <mutante>	 spagewmf: confirmed working?
[02:56:57] <icinga-wm>	 RECOVERY - puppetmaster https on virt0 is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.159 second response time  
[02:57:32] <springle>	 mutante: cool. was jfyi
[02:57:45] <mutante>	 springle: yea, thanks, i really hadnt even seen the merge yet:)
[02:59:19] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Dec 20 02:59:19 UTC 2013
[03:00:06] <icinga-wm>	 PROBLEM - puppetmaster https on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[03:00:08] <spagewmf>	 mutante I thought it was OK but I tried to login again and that's hanging or very slow
[03:00:16] <mutante>	 springle: oh yea, i figured you might be interested because its ui for mkquery digest:) meant you anyways as one reviewer on it
[03:01:20] <spagewmf>	 mutante ^ also virt0 complaint above.
[03:01:23] <mutante>	 spagewmf: i think it's just catching up a bit, appears relatively normal to me
[03:01:36] <mutante>	 i can click around etc
[03:01:57] <icinga-wm>	 RECOVERY - puppetmaster https on virt0 is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.166 second response time  
[03:01:59] <mutante>	 hmm... yea, icinga .. grr
[03:02:31] <mutante>	 there can be a little delay there on the way to icinga-wm
[03:02:39] <mutante>	 as long as it doesnt keep doing this all the time now
[03:03:03] <spagewmf>	 mutante I'm in, yup was just very slow.  Thanks
[03:03:22] <mutante>	 spagewmf: it might also just be that so many users are running stuff now
[03:03:29] <mutante>	 or the puppetmaster tries a whole bunch at a time
[03:06:46] <mutante>	 spagewmf: root cause
[03:06:50] <mutante>	         from /usr/lib/phusion_passenger/passenger-spawn-server:61
[03:06:55] <mutante>	 *** Exception Errno::EPIPE in Passenger RequestHandler (Broken pipe) (process 23592):
[03:07:13] <mutante>	 bug in the passenger module it appears
[03:11:22] <spagewmf>	 wikitech uses Phusion Passenger unlike our other MW servers?
[03:11:28] <mutante>	 springle: fyi, i would say blame phusionpassenger.com for that
[03:11:30] <spagewmf>	 learn something new every day :)
[03:11:46] <mutante>	 spagewmf: well, it's not a normal wiki, it is labs
[03:11:52] <mutante>	 it has been merged into labsconsole
[03:12:00] <mutante>	 wikitech and labsconsole used to be different
[03:12:02] <mutante>	 but not anymore
[03:12:17] <mutante>	 and we have wikitech-static that reedy linked earlier for that reason
[03:12:24] <mutante>	 that we still have docs when site is down
[03:12:40] <mutante>	 we had that covered before by putting it on 3rd party VM out of band
[03:17:25] <grrrit-wm>	 (03PS1) 10Springle: Debug https://gerrit.wikimedia.org/r/96403 [operations/puppet] - 10https://gerrit.wikimedia.org/r/102874 
[03:20:00] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Debug https://gerrit.wikimedia.org/r/96403 [operations/puppet] - 10https://gerrit.wikimedia.org/r/102874 (owner: 10Springle)
[03:20:36] <grrrit-wm>	 (03CR) 10Dzahn: "oh, yea, it will be just that: class { '::ishmael': , missing :: , did that before and we wondered why it recincludes the ROLE class" [operations/puppet] - 10https://gerrit.wikimedia.org/r/102874 (owner: 10Springle)
[03:22:05] <mutante>	 springle: it would declare itself there
[03:22:09] <mutante>	 without the ::
[03:22:20] <mutante>	 we had it before in another _to module_ change
[03:22:42] <mutante>	 will just try to do role::ishmael where you think it's ishmael itself
[03:23:33] <springle>	 ah ok
[03:23:52] * springle  will just get it working :)

[03:24:00] <mutante>	 awesome:)
[03:26:41] <grrrit-wm>	 (03PS1) 10Springle: Debug https://gerrit.wikimedia.org/r/96403 [operations/puppet] - 10https://gerrit.wikimedia.org/r/102875 
[03:27:52] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Debug https://gerrit.wikimedia.org/r/96403 [operations/puppet] - 10https://gerrit.wikimedia.org/r/102875 (owner: 10Springle)
[03:32:59] <jeremyb>	 mutante: for what? :)
[03:35:04] <mutante>	 jeremyb: code review on private changes
[03:35:36] <jeremyb>	 mutante: uhhhh, idk what to tell you about that...
[03:35:41] <jeremyb>	 mutante: normal process is?
[03:36:10] <mutante>	 jeremyb: RT, git log, mail
[03:36:18] <mutante>	 jeremyb: just thought i heard criticism there
[03:37:01] <jeremyb>	 mutante: no, re approval process. you got referral, explicit approval, approval of the approval, etc. :)
[03:37:13] <jeremyb>	 seemed kinda meta/recursive :)
[03:37:24] <mutante>	 jeremyb: we were just having fun and being serious at the same time:)
[03:37:28] <grrrit-wm>	 (03CR) 10Springle: "Is there a way to get the ssl_cert in properly? puppet said:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/102874 (owner: 10Springle)
[03:37:32] <jeremyb>	 mutante: i figured :)
[03:37:47] <jeremyb>	 mutante: you've seen [[list of lists of lists]]?
[03:38:15] <mutante>	 jeremyb: sounds like a mailing list run by thehelpfulone
[03:38:19] <mutante>	 that explains how to admin lists
[03:38:25] <mutante>	 we should have
[03:38:39] <jeremyb>	 https://en.wikipedia.org/wiki/List_of_lists_of_lists
[03:38:54] <mutante>	 hahaa, nice
[03:39:17] <mutante>	 find one in another language!! go for next level
[03:39:43] <jeremyb>	 i don't see one in de
[03:39:44] <mutante>	 and why arent you already on wikidata editing that there
[03:40:12] <jeremyb>	 i'm not editing it anywhere!
[03:40:26] <mutante>	 :)
[03:42:06] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000  
[03:44:23] <jeremyb>	 https://gdash.wikimedia.org/dashboards/reqerror/ did have a spike recently
[04:37:26] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC  
[04:37:26] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC  
[04:40:06] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000  
[06:07:53] <grrrit-wm>	 (03CR) 10Faidon Liambotis: "What Bryan said, let this be handled on the app side. Also, I'm pretty sure not /everything/ in the cache should be Varied on Cookie (e.g." [operations/puppet] - 10https://gerrit.wikimedia.org/r/102744 (owner: 10Ori.livneh)
[06:19:16] <yurik-road2>	 is mark around by any chance? i'm trying to figure out the current IP ranges for Carolynne
[06:19:45] <paravoid>	 for what?
[13:40:26] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC  
[13:40:26] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC  
[13:42:46] <icinga-wm>	 PROBLEM - Host cp1065 is DOWN: PING CRITICAL - Packet loss = 100%  
[13:46:09] <paravoid>	 again...
[13:46:42] <paravoid>	 hah, exactly the same hour too
[13:46:46] <paravoid>	 as yesterday
[13:52:06] <icinga-wm>	 RECOVERY - Host cp1065 is UP: PING OK - Packet loss = 0%, RTA = 0.20 ms  
[13:52:13] <paravoid>	 !log powercycling cp1065
[13:52:29] <morebots>	 Logged the message, Master
[14:28:55] <hashar>	 !log Jenkins setting http_proxy and https_proxy on lanthanum.eqiad.wmnet to point to carbon.wikimedia.org (eqiad web proxy)
[14:29:11] <morebots>	 Logged the message, Master
[14:34:48] <hashar>	 !log Jenkins removed http_proxy env variables from lanthanum.eqiad.wmnet does not play well with git urls :D
[14:35:05] <morebots>	 Logged the message, Master
[14:59:13] <grrrit-wm>	 (03PS1) 10Nemo bis: Relative path in varnish error message: remove excess / [operations/puppet] - 10https://gerrit.wikimedia.org/r/102945 
[15:00:43] <grrrit-wm>	 (03PS2) 10Nemo bis: Relative path in varnish error message: remove excess / [operations/puppet] - 10https://gerrit.wikimedia.org/r/102945 
[15:00:58] <grrrit-wm>	 (03CR) 10Nemo bis: "Followed up in Ie06c65f665d1d2bcca7e89152ff819da8810d379" [operations/puppet] - 10https://gerrit.wikimedia.org/r/65792 (owner: 10Mark Bergsma)
[15:57:56] <grrrit-wm>	 (03PS3) 10John F. Lewis: Redirect m.wikidata to www.wikidata.org [operations/apache-config] - 10https://gerrit.wikimedia.org/r/101787 
[16:00:33] <grrrit-wm>	 (03PS3) 10John F. Lewis: Redirect kr.wikimedia [operations/apache-config] - 10https://gerrit.wikimedia.org/r/101220 
[16:13:37] <hashar>	 !jenkins visualeditor-doitall
[16:13:38] <wm-bot>	 https://integration.wikimedia.org/ci/job/visualeditor-doitall
[16:18:24] <andrewbogott>	 !log merged a couple of tiny OpenStackManager patches on wikitech
[16:18:41] <morebots>	 Logged the message, Master
[16:20:41] <grrrit-wm>	 (03CR) 10Jeremyb: [C: 04-1] "Also, needs to be ported to the new redirects system if it's ever ready on the DNS/registration side." [operations/apache-config] - 10https://gerrit.wikimedia.org/r/88705 (owner: 10Dzahn)
[16:41:26] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC  
[16:41:26] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC  
[18:44:16] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000  
[19:01:56] <icinga-wm>	 PROBLEM - Host cp1065 is DOWN: PING CRITICAL - Packet loss = 100%  
[19:08:49] <mutante>	 yep, that died, on it
[19:09:34] <mutante>	 !log powercycling cp1065
[19:09:51] <morebots>	 Logged the message, Master
[19:09:52] <mutante>	 !log cp1065 - [19060.904886] BUG: scheduling while atomic: kworker/11:0/27073/0x00000200
[19:10:10] <morebots>	 Logged the message, Master
[19:12:26] <icinga-wm>	 RECOVERY - Host cp1065 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms  
[19:12:33] <jeremyb>	 mutante: i think that's at least the 3rd time in 2 days?
[19:12:52] <jeremyb>	 and don't see it in RT
[19:13:10] <mutante>	 bonus, services upstart vs. salt
[19:13:12] <mutante>	 Rather than invoking init scripts through /etc/init.d, use the service(8)
[19:13:15] <mutante>	 utility, e.g. service S20salt-minion start
[19:13:18] <mutante>	 initctl: Unknown job: S20salt-minion
[19:13:59] <mutante>	 jeremyb: not seeing it in RT may explain i didnt see it either recently at least
[19:14:19] <jeremyb>	 mutante: did you try salt-minion instead of S20salt-minion?
[19:15:03] <jeremyb>	 mutante: oh, whoops. actually that's 3x in <1 day
[19:15:07] <mutante>	 jeremyb: i dont think i need to do anything more, salt-minion is actually running
[19:15:19] <jeremyb>	 10:19:15 13:52:12 19:09:34
[19:15:27] <mutante>	 so it is more annoyance in the order/output when it comes up
[19:34:36] <icinga-wm>	 PROBLEM - Host cp1065 is DOWN: PING CRITICAL - Packet loss = 100%  
[19:35:35] <apergos>	 that was fast
[19:39:16] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000  
[19:42:26] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC  
[19:42:26] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC  
[20:05:26] <bblack>	 that cp1065 crash has got to be hardware related at this point, imho
[20:24:57] <grrrit-wm>	 (03PS1) 10Jgreen: add cname for giantrabbit civi testing on lutetium [operations/dns] - 10https://gerrit.wikimedia.org/r/102986 
[20:27:28] <grrrit-wm>	 (03CR) 10Jgreen: [C: 032 V: 031] add cname for giantrabbit civi testing on lutetium [operations/dns] - 10https://gerrit.wikimedia.org/r/102986 (owner: 10Jgreen)
[20:33:03] <mutante>	 bah, tons of .exe on my computer because from a disk somebody wrote with windows and rsync is now busy on syncing his trashcan and system restore as well :p
[20:34:31] <mutante>	 bblack: apergos oh, it's down again? was away for a couple minutes
[20:34:39] <mutante>	 do we still care now to bring it back?
[20:34:58] <apergos>	 good question
[20:35:36] <apergos>	 those that are actively deubgging the issue should weigh in I guess
[20:35:38] <jeremyb>	 well or shut it down completely?
[20:36:00] <jeremyb>	 or bring it back but leave varnish off? and see if it dies with no load? :-)
[20:36:11] <mutante>	 pooled or unpooled?
[20:36:17] <paravoid>	 can you bring it up with 3.2?
[20:36:21] <paravoid>	 just in case
[20:36:36] <mutante>	 ok
[20:38:48] <mutante>	 !log powercycled cp1065 again, will upgrade kernel this time
[20:39:06] <morebots>	 Logged the message, Master
[20:40:06] <icinga-wm>	 RECOVERY - Host cp1065 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms  
[20:40:16] <mutante>	 paravoid: all pending upgrades ok or do you want to make sure it is JUST kernel
[20:40:23] <mutante>	 didnt get to see how many yet
[20:40:29] <mutante>	 its coming back up slowly
[20:41:08] <paravoid>	 just kernel please
[20:42:06] <icinga-wm>	 PROBLEM - Varnish HTTP text-backend on cp1065 is CRITICAL: Connection refused  
[20:42:43] <mutante>	 paravoid: on it, rebooting again, do i need to look at depooling at all?
[20:43:18] <paravoid>	 nah
[20:43:45] <mutante>	 !cp1065 - was 3.11.0-13-generic, rebooting for 3.2.0-57-generic
[20:43:56] <paravoid>	 thank you
[20:43:57] <mutante>	 !log cp1065 - was 3.11.0-13-generic, rebooting for 3.2.0-57-generic
[20:44:08] <paravoid>	 it's pretty late here, sorry for not doing it myself
[20:44:12] <morebots>	 Logged the message, Master
[20:44:14] <mutante>	 no problem 
[20:45:46] <icinga-wm>	 PROBLEM - Host cp1065 is DOWN: PING CRITICAL - Packet loss = 100%  
[20:46:21] <mutante>	 it worked
[20:47:06] <icinga-wm>	 RECOVERY - Host cp1065 is UP: PING OK - Packet loss = 0%, RTA = 0.41 ms  
[20:47:07] <mutante>	 Linux cp1065 3.11.0-13-generic #20~precise2-Ubuntu SMP Thu Oct 24 21:04:34 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
[20:47:12] <mutante>	 and now we shall wait
[20:47:19] <paravoid>	 that's 3.11
[20:47:28] <mutante>	 damn it is
[20:48:44] <mutante>	 but before i get to fix it even
[20:48:46] <mutante>	 [  132.718522] BUG: soft lockup - CPU#1 stuck for 22s! [swapper/1:0]
[20:49:10] <paravoid>	 really?
[20:49:56] <icinga-wm>	 PROBLEM - Host cp1065 is DOWN: PING CRITICAL - Packet loss = 100%  
[20:50:09] <mutante>	 sure i just need to updategrub.. but already hard to use
[20:50:20] <jeremyb>	 maybe you should invest in a roller coaster?
[20:55:56] <mutante>	 ok, it has one more chance now. updated grub
[20:56:49] <Reedy>	 just take it out back and shoot it
[20:59:07] <icinga-wm>	 RECOVERY - Host cp1065 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms  
[21:00:39] <mutante>	 paravoid: wth, what else besides installing package, and update-grub ..it still booted into 3.11 .. and fwiw it still just gives me a few seconds of time to change things before the next issue
[21:00:56] <mutante>	 so didnt get to grub conf again :p
[21:01:46] <icinga-wm>	 PROBLEM - Host cp1065 is DOWN: PING CRITICAL - Packet loss = 100%  
[21:01:46] <jeremyb>	 mutante: disable varnish from starting by itself, fix kernel, reenable varnish?
[21:02:30] <mutante>	 jeremyb: worth a try,thx
[21:06:14] <icinga-wm>	 ACKNOWLEDGEMENT - DPKG on cp1065 is CRITICAL: Timeout while attempting connection daniel_zahn kernel bug and/or hardware issue
[21:06:14] <icinga-wm>	 ACKNOWLEDGEMENT - Disk space on cp1065 is CRITICAL: Timeout while attempting connection daniel_zahn kernel bug and/or hardware issue
[21:06:14] <icinga-wm>	 ACKNOWLEDGEMENT - NTP on cp1065 is CRITICAL: NTP CRITICAL: No response from NTP server daniel_zahn kernel bug and/or hardware issue
[21:06:14] <icinga-wm>	 ACKNOWLEDGEMENT - RAID on cp1065 is CRITICAL: Timeout while attempting connection daniel_zahn kernel bug and/or hardware issue
[21:06:14] <icinga-wm>	 ACKNOWLEDGEMENT - SSH on cp1065 is CRITICAL: Connection timed out daniel_zahn kernel bug and/or hardware issue
[21:06:15] <icinga-wm>	 ACKNOWLEDGEMENT - Varnish HTCP daemon on cp1065 is CRITICAL: Timeout while attempting connection daniel_zahn kernel bug and/or hardware issue
[21:06:15] <icinga-wm>	 ACKNOWLEDGEMENT - Varnish HTTP text-backend on cp1065 is CRITICAL: Connection timed out daniel_zahn kernel bug and/or hardware issue
[21:06:16] <icinga-wm>	 ACKNOWLEDGEMENT - Varnish HTTP text-frontend on cp1065 is CRITICAL: Connection timed out daniel_zahn kernel bug and/or hardware issue
[21:06:16] <icinga-wm>	 ACKNOWLEDGEMENT - Varnish traffic logger on cp1065 is CRITICAL: Timeout while attempting connection daniel_zahn kernel bug and/or hardware issue
[21:06:17] <icinga-wm>	 ACKNOWLEDGEMENT - puppet disabled on cp1065 is CRITICAL: Timeout while attempting connection daniel_zahn kernel bug and/or hardware issue
[21:06:42] <Nemo_bis>	 uh, Daniel also is an NTP server, who knew
[21:06:49] <Nemo_bis>	 better than Kant
[21:08:06] <icinga-wm>	 RECOVERY - Host cp1065 is UP: PING OK - Packet loss = 0%, RTA = 0.99 ms  
[21:10:34] <mutante>	 jeremyb: not fast enough 2/3 services or so:)
[21:10:46] <icinga-wm>	 PROBLEM - Host cp1065 is DOWN: PING CRITICAL - Packet loss = 100%  
[21:11:33] <jeremyb>	 mutante: you can get to console? boot it up single user and run your grub update/etc. there?
[21:14:36] <mutante>	 jeremyb: yea, init 1 after a sandwich or so:)
[21:15:57] <jeremyb>	 mutante: i meant straight to single user on boot. append ' s' to kernel params
[21:16:37] <mutante>	 yea, either or, it usually gives me enough and was already coming up just now
[21:17:06] <icinga-wm>	 RECOVERY - Host cp1065 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms  
[21:19:06] <icinga-wm>	 PROBLEM - Varnish HTCP daemon on cp1065 is CRITICAL: Connection refused by host  
[21:19:56] <icinga-wm>	 PROBLEM - puppet disabled on cp1065 is CRITICAL: Connection refused by host  
[21:20:06] <icinga-wm>	 PROBLEM - SSH on cp1065 is CRITICAL: Connection refused  
[21:20:06] <icinga-wm>	 PROBLEM - Disk space on cp1065 is CRITICAL: Connection refused by host  
[21:20:06] <icinga-wm>	 PROBLEM - DPKG on cp1065 is CRITICAL: Connection refused by host  
[21:20:06] <icinga-wm>	 PROBLEM - RAID on cp1065 is CRITICAL: Connection refused by host  
[21:26:07] <grrrit-wm>	 (03PS2) 10Yurik: Fixed a major bug with evaluation of netmapper usage [operations/puppet] - 10https://gerrit.wikimedia.org/r/102887 
[21:26:34] <grrrit-wm>	 (03PS6) 10Yurik: Handle HTTPS for Zero traffic [operations/puppet] - 10https://gerrit.wikimedia.org/r/102316 
[21:35:33] <grrrit-wm>	 (03PS2) 10Ebrahim: Use local Wiki.png for Persian Wikipedia Logo [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102454 
[21:37:48] <mutante>	 !log cp1065 - currently init 1 sitting at maintenance but not powered down, i will poke later or anyone can
[21:38:06] <morebots>	 Logged the message, Master
[21:47:11] <yurik>	 greg-g, when you say ZERO DEPLOYS on dec 23rd, are you saying we are the only ones that should deploy that week?  thanks!
[21:50:04] <greg-g>	 yurik: :P
[21:50:16] <greg-g>	 all zero, all week
[21:55:58] <Nemo_bis>	 all the cluster HDDs will be zero'ed
[21:57:12] <greg-g>	 so many interpretations :)
[22:06:59] <grrrit-wm>	 (03CR) 10Odder: [C: 031] Use local Wiki.png for Persian Wikipedia Logo [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/102454 (owner: 10Ebrahim)
[22:07:47] <grrrit-wm>	 (03PS1) 10Nemo bis: [Planet] Update wikimedia.fi URL [operations/puppet] - 10https://gerrit.wikimedia.org/r/103047 
[22:29:22] <yurik>	 anyone has any idea why Bug58676.log is still empty?  there are plenty of them in fatalmonitor:  /usr/local/apache/common-local/php-1.23wmf7/includes/Message.php on line 822
[22:43:26] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:28:53 PM UTC  
[22:43:26] <icinga-wm>	 PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Last successful Puppet run was Wed 18 Dec 2013 10:29:11 PM UTC  
[22:44:25] <mutante>	 ok, i'm out of IRC for now, if there is anything for me, use RT please, it's RT duty, not also IRC duty:) or memoserv is fine as stated before.. cya
[22:55:09] <ori-l>	 Coren: reminder re: <https://gerrit.wikimedia.org/r/#/c/101793/>. If you're already in Christmas mode, it can wait until January.
[22:55:59] <Coren>	 I'm not in vacation mode yet, but that will require a bit of quiet time and concentration so I'll probably do it between xmas and the new year.
[22:56:25] <ori-l>	 Totally cool, thanks very much.