[00:02:25] <AaronSchulz>	 paravoid: 1.8M commons temp objects...graet
[00:02:27] <AaronSchulz>	 *great
[00:02:36] <AaronSchulz>	 this cleanup script will take ages
[00:03:00] <AaronSchulz>	 it would be faster to backport a change to make it do batching, sync, and restart
[00:03:12] <grrrit-wm>	 (03PS1) 10Dzahn: remove gurvin and gurvin.mgmt, decom [operations/dns] - 10https://gerrit.wikimedia.org/r/94448 
[00:08:13] <grrrit-wm>	 (03PS1) 10Dzahn: remove gurvin from dhcp, dsh and site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/94449 
[00:09:21] <grrrit-wm>	 (03CR) 10Dzahn: "see comments on the linked ticket. just wondering why it still had certificates::wmf_ca on it." [operations/puppet] - 10https://gerrit.wikimedia.org/r/94449 (owner: 10Dzahn)
[00:32:51] <logmsgbot>	 !log aaron synchronized php-1.23wmf2/maintenance/cleanupUploadStash.php  '01da29d3ce23cea02df3fc975d6d51bfbc50a222'
[00:33:11] <morebots>	 Logged the message, Master
[00:45:30] <grrrit-wm>	 (03PS1) 10Dzahn: remove osm-web1-4 and mgmt [operations/dns] - 10https://gerrit.wikimedia.org/r/94457 
[00:46:05] <logmsgbot>	 !log aaron synchronized php-1.23wmf3/maintenance
[00:46:20] <morebots>	 Logged the message, Master
[01:11:17] <logmsgbot>	 !log aaron synchronized php-1.23wmf2/includes/filebackend  '2afdc066f52b54faf63f9c980f7ef6a7841dd094'
[01:11:36] <morebots>	 Logged the message, Master
[01:11:47] <logmsgbot>	 !log aaron synchronized php-1.23wmf3/includes/filebackend  '2afdc066f52b54faf63f9c980f7ef6a7841dd094'
[01:12:04] <morebots>	 Logged the message, Master
[01:16:51] <logmsgbot>	 !log gwicke synchronized php-1.23wmf2/extensions/Parsoid/php
[01:17:05] <morebots>	 Logged the message, Master
[01:19:30] <logmsgbot>	 !log aaron synchronized php-1.23wmf3/maintenance  'c674d4c90c40d18187d893ed7a7ea94f43a1a624'
[01:19:50] <morebots>	 Logged the message, Master
[01:19:53] <gwicke>	 and parsoid jobs are back: https://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&s=by+name&c=Parsoid%2520eqiad&tab=m&vn=
[01:19:59] <logmsgbot>	 !log aaron synchronized php-1.23wmf2/maintenance  'c674d4c90c40d18187d893ed7a7ea94f43a1a624'
[01:20:18] <morebots>	 Logged the message, Master
[01:23:15] <logmsgbot>	 !log gwicke synchronized php-1.23wmf3/extensions/Parsoid/php
[01:23:30] <morebots>	 Logged the message, Master
[01:26:16] <^d>	 Um, what's going on with the apis?
[01:26:17] <^d>	 http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=API%20application%20servers%20eqiad&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2&st=1383960343&g=network_report&z=large
[01:26:32] <^d>	 (network, last hour)
[01:27:01] <gwicke>	 ^d: parsoid back in action
[01:27:07] <^d>	 Ah, gotcha :)
[01:27:18] <^d>	 Doesn't look dangerous, just happened to notice.
[01:27:21] <^d>	 :p
[01:27:30] <gwicke>	 99% of our load comes from our the job runner which I just fixed
[01:27:43] <^d>	 *nod*
[01:29:38] <gwicke>	 we do a lot of requests for content, but each of those is quite cheap for the api
[02:03:45] <logmsgbot>	 !log LocalisationUpdate completed (1.23wmf2) at Sat Nov  9 02:03:45 UTC 2013
[02:04:05] <morebots>	 Logged the message, Master
[02:04:56] <logmsgbot>	 !log LocalisationUpdate completed (1.23wmf3) at Sat Nov  9 02:04:55 UTC 2013
[02:05:13] <morebots>	 Logged the message, Master
[02:10:54] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Nov  9 02:10:54 UTC 2013
[02:11:13] <morebots>	 Logged the message, Master
[02:28:53] <grrrit-wm>	 (03PS1) 10Jforrester: Create visualeditor-default.dblist to simplify config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94471 
[02:28:53] <grrrit-wm>	 (03PS1) 10Jforrester: Remove cruft from wmgVisualEditorDisableForAnons no longer needed [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94472 
[02:32:51] <grrrit-wm>	 (03PS5) 10Jforrester: Enable VisualEditor on board, collab, office and wikimaniateam wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/93780 
[03:00:24] <Reedy>	 ori-l: About
[03:00:25] <Reedy>	 ?
[03:01:00] <Reedy>	 Or someone else from ops..
[03:01:21] <Reedy>	 Could someone add me as a projectadmin to the tool labs project please?
[03:42:37] <icinga-wm>	 PROBLEM - Puppet freshness on sq48 is CRITICAL: No successful Puppet run in the last 10 hours  
[04:57:49] <grrrit-wm>	 (03PS2) 10TTO: Renamed $wmfConfigDir to $wmgConfigDir in mediawiki-config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94354 (owner: 10Arav93)
[04:57:53] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Renamed $wmfConfigDir to $wmgConfigDir in mediawiki-config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94354 (owner: 10Arav93)
[05:35:54] <logmsgbot>	 !log aaron synchronized php-1.23wmf2/includes/filebackend/SwiftFileBackend.php  '519883bd67c960998e8cb2eb0cda52f3764192f7'
[05:36:11] <morebots>	 Logged the message, Master
[05:37:47] <logmsgbot>	 !log aaron synchronized php-1.23wmf3/includes/filebackend/SwiftFileBackend.php  '519883bd67c960998e8cb2eb0cda52f3764192f7'
[05:38:05] <morebots>	 Logged the message, Master
[05:46:09] <grrrit-wm>	 (03PS3) 10TTO: Clean up wgSiteName in InitialiseSettings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86418 
[05:48:15] <Aaron|home>	 paravoid: https://blueprints.launchpad.net/swift/+spec/pool-memcache-connections sounds interesting
[05:55:34] <Aaron|home>	 https://bugs.launchpad.net/swift/+bug/1229142 is fun too (not that it effects us)
[06:10:16] <apergos>	 Reedy: still around?
[06:11:43] <apergos>	 ah it looks like someone already added you
[06:13:20] <Aaron|home>	 apergos: maybe you can merge https://gerrit.wikimedia.org/r/#/c/94480/1 :)
[06:13:27] <apergos>	 hahaha
[06:13:44] * Aaron|home  already backported it

[06:14:07] <apergos>	 ok but maybe you can look at https://gerrit.wikimedia.org/r/#/c/91213/ :-D
[06:15:31] <Aaron|home>	 I see it in the db conf, ok
[06:16:26] <apergos>	 lgmt merged
[06:16:35] <apergos>	 *tm
[06:20:12] <greg-g>	 oh man, more trademark bs?
[06:20:40] <apergos>	 which thing is this?
[06:22:04] <grrrit-wm>	 (03CR) 10ArielGlenn: [C: 031] fix pdf servers in dsh groups [operations/puppet] - 10https://gerrit.wikimedia.org/r/94307 (owner: 10Dzahn)
[06:22:30] <greg-g>	 apergos: you :)
[06:22:35] <greg-g>	 apergos: the "*tm"
[06:22:41] <greg-g>	 sorry, bad late-ish night joke
[06:22:43] <apergos>	 :-P
[06:22:48] <apergos>	 it is, and it's early morning
[06:23:01] <apergos>	 I"m awake enough to review (a little) but not awake enough for that!
[06:24:12] <apergos>	 nt awake enough to review bugz module though... later or monday
[07:08:16] <jeremyb>	 RobH: 2 ideas from the people sitting next to me: archivists and whistleblowers
[07:22:28] <apergos>	 then we will get to have he debate about whether a misc host named 'snowden' is too obvious a target for script kiddies
[07:22:57] <apergos>	 archivists might overlap some with encyclopedians 
[07:34:58] <p858snake|l>	 keep the private wiki dumps on snowden >.>
[07:40:19] <apergos>	 FAT CHANCE
[07:40:47] <greg-g>	 no fun
[07:41:04] <apergos>	 dang straight
[08:01:04] <icinga-wm>	 PROBLEM - Puppet freshness on sq80 is CRITICAL: No successful Puppet run in the last 10 hours  
[08:02:52] <p858snake|l>	 snowden assange and manning could be the search boxes...
[08:47:29] <grrrit-wm>	 (03CR) 10Nemo bis: "(See steps at <https://www.mediawiki.org/wiki/Gerrit/Advanced_usage#Build_failed_due_to_merge_conflict>.)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94354 (owner: 10Arav93)
[08:51:53] <paravoid>	 morning
[08:52:48] <Nemo_bis>	 mogge
[08:53:06] <paravoid>	 springle: hey
[08:53:24] <paravoid>	 can I bother you for 3' to give you an update on something? :)
[08:55:10] <springle>	 paravoid: sure (apergos started already)
[08:56:04] <Aaron|home>	 hello
[08:56:20] <paravoid>	 we had an outage yesterday
[08:56:26] <paravoid>	 between 18:17 UTC and 18:31 UTC
[08:56:56] <paravoid>	 s1 shard had a huge load spike, along with number of rows, open transactions, rows select
[08:57:02] <paravoid>	 cause is still unknown
[08:57:16] <paravoid>	 we lost db1050 in the process, it just became completely unresponsive
[08:57:44] <paravoid>	 I verified that load started increasing across the databases before we lost db1050, though, so it's likely an effect, not a cause
[08:57:59] <paravoid>	 (although it then obviously made the situation even worse)
[08:58:19] <paravoid>	 we powercycled db1050 and then something wrong with LVM and hasn't come up properly yet -- I depooled it from mw10NN
[08:59:02] <paravoid>	 I digged through ishmael, found a few outliers and pointed them out, one of them was fixed but it doesn't look related to this
[08:59:17] <paravoid>	 ori also found an uploadwizard slow query and turned it off
[08:59:38] <paravoid>	 no hard evidence it was the cause of this
[09:00:36] <springle>	 interesting that unpurged txns on db1056 master spiked over that time. something wrote heavily to it (which binlog may show) or else ran slow SELECT causing purge lag
[09:01:31] <paravoid>	 oh and https://ishmael.wikimedia.org/more.php?hours=24&host=db1052&checksum=2279768281867546317 was the other one that I found that looked nasty
[09:01:47] <paravoid>	 at one point, it was 9% of queries, 90% of time for past 1 hour
[09:02:05] <paravoid>	 and it does look elevated since yesterday
[09:02:33] <paravoid>	 and during a very similar time to the outage (20' before)
[09:02:35] <springle>	 yes LogPager has been a problem for a while. bug open, and believe it or not it's actually better than it was
[09:02:51] <Aaron|home>	 that's what I said ;)
[09:03:14] <paravoid>	 Aaron|home: indeed, sorry, I didn't mean to disregard your comment, just dumping all the information to sean :)
[09:03:47] <springle>	 paravoid: thank you :)
[09:04:28] <paravoid>	 I was thinking of writing the incident report, but it's all so unclear that it'd look ridiculous
[09:04:44] <Aaron|home>	 paravoid: heh, I was just talking about the "actually better than it was" bit
[09:04:55] <paravoid>	 "something happened somewhere and we crashed"
[09:05:09] <springle>	 were the other layers flapping because of db failures, or in advance of it?
[09:05:37] <paravoid>	 my investigation showed a cascading failure starting from databases
[09:05:45] <springle>	 ok
[09:06:02] <paravoid>	 i.e. apache requests piled up waiting for dbs and produced a site-wide (all shards) failure
[09:06:09] <paravoid>	 both regular appservers and api appservers
[09:11:03] <springle>	 there was definitely a write spike on the S1 master db1056 at that time
[09:12:27] <paravoid>	 how do you figure?
[09:13:08] <paravoid>	 can you point me at what you're looking?
[09:13:18] <paravoid>	 so that I know for next time ;)
[09:15:12] <paravoid>	 Aaron|home: btw, I got a tentative window from greg-g next thursday, to switch the Swift master
[09:16:13] <springle>	 paravoid: http://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&title=mysql_innodb_transactions_unpurged&mreg[]=^mysql_innodb_transactions_unpurged%24&hreg[]=db1056&aggregate=1&hl=db1056.eqiad.wmnet|MySQL+eqiad
[09:16:36] <springle>	 two spikes, first one about the right time?
[09:16:48] <paravoid>	 yes
[09:17:04] <paravoid>	 when there's doubt, I use the CSV button on top of the graph
[09:17:13] <paravoid>	 then go the timestamp and manually read it
[09:17:39] <paravoid>	 (what I'd give for interactive graphs you can hover over and they'd tell you the timestamp & value)
[09:19:32] <springle>	 similar metric, but may also indicate slow SELECT on master: http://ganglia.wikimedia.org/latest/graph.php?r=day&z=xlarge&title=mysql_innodb_history_list&mreg[]=^mysql_innodb_history_list%24&hreg[]=db1056&aggregate=1&hl=db1056.eqiad.wmnet|MySQL+eqiad
[09:21:26] <springle>	 looks like ganglia didn't get data from many S1 boxes around that time, but innodb hist list length takes time to reduce after a spike, so we can see the aftermath
[09:33:32] <icinga-wm>	 PROBLEM - MySQL InnoDB on db1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[09:34:21] <icinga-wm>	 RECOVERY - MySQL InnoDB on db1002 is OK: OK longest blocking idle transaction sleeps for 0 seconds  
[09:59:54] <paravoid>	 springle: are you still investigating? I want to play with ganglia a bit and I may make the web interface unavailable in the process
[10:00:00] <paravoid>	 but I can wait if you are
[10:00:16] <springle>	 paravoid: i am, but that's fine. (am in binlogs)
[10:08:42] <paravoid>	 !log upgraded ganglia-web to 3.5.10
[10:09:02] <morebots>	 Logged the message, Master
[10:31:14] <Nemo_bis>	 paravoid: link to new shiny features? :)
[10:32:48] <Nemo_bis>	 hmm several new small useful things I see
[10:37:20] <paravoid>	 yeah, and breaking other crap
[10:39:57] <grrrit-wm>	 (03PS3) 10Ori.livneh: Add Graphite module & role [operations/puppet] - 10https://gerrit.wikimedia.org/r/92271 
[10:40:22] <paravoid>	 grrr
[10:40:31] <paravoid>	 the ganglia-web code is so horrible
[10:43:52] <ori-l>	 too tired to write a proper commit message
[10:44:06] <ori-l>	 before i go to bed, here's one i thought you might enjoy, paravoid
[10:44:16] <ori-l>	 the actual, no-joke output of 'uwsgi --help': https://dpaste.de/N8js/raw/
[10:44:40] <paravoid>	 hahahaha
[10:45:05] <paravoid>	     --spooler-harakiri               *** UNDOCUMENTED OPTION ***
[10:45:06] <paravoid>	     --mule-harakiri                  *** UNDOCUMENTED OPTION ***
[10:45:10] <paravoid>	 rofl
[10:45:34] <ori-l>	 yeah, i am actually laughing too, i mean -- seriously
[10:45:55] <ori-l>	 --zerg-server                    *** UNDOCUMENTED OPTION ***
[10:46:20] <ori-l>	 anyways, good night
[10:46:36] <springle>	 paravoid: looking throught dberror.log during the outage; odd that connection issues are logged for many different mysql boxes, including external storage hosts(!?), by no means just S1 or wikidata, around the time
[10:47:36] <paravoid>	 bye ori-l, thanks for the laugh
[10:47:40] <springle>	 how would a purely S1 spike or outage would affect all the others?
[10:47:45] <paravoid>	 springle: at what times where those though?
[10:48:08] <springle>	 between 18:15 and 18:29
[10:48:12] <paravoid>	 we do know that apaches went haywire at some point, possibly due to an effect of the s1 overload
[10:48:14] <springle>	 same period
[10:48:17] <paravoid>	 requests piled etc.
[10:48:41] <paravoid>	 so if it's the middle of the outage, it could be explained this way
[10:48:50] <springle>	 earliest db errors logged in that period were not S1 hosts...
[10:48:53] <springle>	 hmm ok
[10:48:54] <paravoid>	 if it's as early as 18:15, though, then it won't explain it, no
[10:53:29] <springle>	 paravoid: did anything happen to ganglia in that period?
[10:53:44] <springle>	 or to stats aggregation in general
[10:53:58] <paravoid>	 not afaik
[10:57:37] <jeremyb>	 springle: i'm thinking about using externallinks in labs. just wondering if that's one that will be getting a primary key or modified timestamp. (you were doing that for other tables, right?)
[10:57:51] <jeremyb>	 or maybe you have extra columns already and i just can't see them
[10:59:03] <springle>	 jeremyb: externallinks is getting a PK now. it's about half done. a few of the largest wikis remain to migrate
[10:59:30] <jeremyb>	 springle: woot. any particular way i should track that? !log ?
[11:00:30] <springle>	 jeremyb: i !log-ged the start of the process, but not every wiki
[11:00:51] <jeremyb>	 ok, maybe i'll just look or ask again in a bit
[11:01:28] <jeremyb>	 i guess the new PK is guaranteed higher (it's an insert + autoincrement) if it's removed and readded
[11:01:42] <jeremyb>	 so that tells you chronology but not actual time
[11:01:54] <springle>	 it's going to be days, likely a week for enwiki, to avoid lagging slaves. and I'm only running it while I can be around to watch dbtree ;)
[11:02:07] <jeremyb>	 right
[11:02:28] <jeremyb>	 i suppose i could start with a smaller one while i get it working. enwiki was the original target
[11:03:09] <springle>	 once we have PK we can do online schema migrations much faste. no technical problem with having an el_stamp field. is there a bug or gerrit commit for it already?
[11:03:47] <jeremyb>	 no. haven't even discussed it with anyone. i thought maybe you were giving it to me for free ;)
[11:03:55] <springle>	 heh
[11:03:59] <springle>	 no such luck
[11:04:28] <jeremyb>	 a related use case I might address with this: https://bugzilla.wikimedia.org/32026
[11:04:35] <jeremyb>	 but not the original reason i was thinking of it
[11:26:14] <jeremyb-phone>	 springle: so is there 1 or 2 DBs already finished I can test against? (externallinks PK)
[11:30:31] <springle>	 jeremyb-phone: many. bgwiki for eg. but not in labs yet i suspect, as it would require rebuilding views
[11:31:00] <jeremyb-phone>	 this I can test ;)
[11:39:22] <jeremyb>	 springle: you're right, it's in bgwiki but not bgwiki_p (not the view)
[11:44:00] <springle>	 paravoid: https://ishmael.wikimedia.org/more.php?host=db1052&hours=24&checksum=10298050210183520632
[11:46:07] <springle>	 was that one you guys already discussed?
[11:50:15] <paravoid>	 springle: yes
[11:50:32] <paravoid>	 I pointed it out, noone could tell me what it does, I left it at that
[11:52:57] <springle>	 i think you were right to wonder about it
[11:55:34] <springle>	 huh.. the sample ishmael captured shows LIMIT 86991,2
[11:56:02] <springle>	 another one LIMIT 86779,2
[11:56:05] <springle>	 etc
[12:08:03] <springle>	 it's increased in frequency in the last week or so too. several short bursts of many concurrent instances each day
[13:07:19] <grrrit-wm>	 (03CR) 10Helder.wiki: [C: 031] Make VisualEditor namespaces extend, not replace, default [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94420 (owner: 10Jforrester)
[13:42:49] <icinga-wm>	 PROBLEM - Puppet freshness on sq48 is CRITICAL: No successful Puppet run in the last 10 hours  
[13:46:30] <springle>	 !log starting pt-kill daemon on S1 slaves for SpecialAllpages::showToplevel queries > 30s
[13:46:46] <morebots>	 Logged the message, Master
[14:30:42] <hexmode>	 anyone around to help with RT/download.wm.o
[14:31:11] <hexmode>	 ?
[14:31:16] <hexmode>	 https://rt.wikimedia.org/SelfService/Display.html?id=6270
[14:33:06] <grrrit-wm>	 (03PS1) 10Arav93: Renamed $wmf* to $wmg* to improve consistency [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94503 
[14:33:08] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Renamed $wmf* to $wmg* to improve consistency [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94503 (owner: 10Arav93)
[14:34:08] <grrrit-wm>	 (03CR) 10Anomie: [C: 04-1] "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94354 (owner: 10Arav93)
[14:38:50] <grrrit-wm>	 (03CR) 10Anomie: "Also, this appears to be roughly identical to I5a18a70a. One or the other should be abandoned," [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94354 (owner: 10Arav93)
[14:38:53] <grrrit-wm>	 (03CR) 10Anomie: [C: 04-1] "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94503 (owner: 10Arav93)
[15:33:19] <grrrit-wm>	 (03Abandoned) 10Arav93: Renamed $wmf* to $wmg* to improve consistency [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94503 (owner: 10Arav93)
[15:38:06] <grrrit-wm>	 (03PS3) 10Arav93: Renamed $wmfConfigDir to $wmgConfigDir in mediawiki-config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94354 
[15:38:13] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Renamed $wmfConfigDir to $wmgConfigDir in mediawiki-config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94354 (owner: 10Arav93)
[16:38:58] <Nemo_bis>	 Thehelpfulone: didn't we have a bug report about timeouts on mailman when dealing with spam/moderation queue?
[16:39:22] <Nemo_bis>	 I remember eia and/or Fae complaining about it
[16:39:43] <Nemo_bis>	 oh well, Nikerabbit will file another one
[16:40:35] <Nemo_bis>	 Nikerabbit: I think it's unrelated
[16:40:39] <Nemo_bis>	 Dario says "FYI there’s a temporary issue that prevents people from unsubscribing from this list. I notified ops."
[16:40:45] <Nemo_bis>	 that's probably your problem
[16:41:30] <Nikerabbit>	 Nemo_bis: where when what list
[17:08:02] <grrrit-wm>	 (03PS1) 10Nemo bis: Make current $wgNoFollowLinks = true explicit [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94508 
[17:42:07] <awight>	 Hi, I'd like to parse through server debug logs for "edit conflict" stuffs.  Is there a policy around how to pursue this?  And, for how long are debug logs archived?
[17:43:06] <Reedy>	 awight: You'll need to start collecting them first
[17:44:06] <awight>	 Reedy: meaning just that they are split by appserver?
[17:53:06] <Nemo_bis>	 awight: do you know https://wikitech.wikimedia.org/wiki/Logs ?
[17:53:23] <grrrit-wm>	 (03CR) 10Aklapper: [C: 04-1] "Patch is completely broken with lots of ">>>>>>>" noise (probably due to a missing rebase)." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/94354 (owner: 10Arav93)
[17:53:36] <jeremyb>	 better link for hexmode https://rt.wikimedia.org/Ticket/Display.html?id=6270 :)
[17:54:02] <jeremyb>	 i was momentarily confused and thought that RT had somehow been "upgraded" :P
[17:54:16] <awight>	 Nemo_bis: interesting, thanks for that link.  I'll poke around to see if wfDebug goes into one of these buckets.
[17:54:46] <hexmode>	 jeremyb: tyvm
[17:55:51] <Nemo_bis>	 awight: is that log supported in MediaWiki? if yes you just need to check the config to see if it's enabled; and if yes, AFAICS every log group has its own file
[17:57:55] <jeremyb>	 Nemo_bis: i think he was asking if they are split into different files (or stored on different machines) per apache box
[17:58:16] <Nemo_bis>	 that's the old question, which the page should have answered
[17:59:11] <jeremyb>	 Nemo_bis: sure, just distinguishing from "log group has its own file"
[18:01:41] <icinga-wm>	 PROBLEM - Puppet freshness on sq80 is CRITICAL: No successful Puppet run in the last 10 hours  
[18:04:03] <awight>	 Nemo_bis: ah, so it seems that wfDebugLog() dumps into those buckets, but I still don't see what happens to plain old wfDebug() messages
[18:07:54] <Nemo_bis>	 awight: they're not logged
[18:08:17] <awight>	 boo.  K, thanks for the confirmation
[18:08:38] <jeremyb>	 awight: do you have access to flourine?
[18:08:52] <Nemo_bis>	 that's my understanding, it's all hearsay for me :)
[18:10:40] <awight>	 jeremyb: yep
[18:13:21] <awight>	 currently causing a processor to die of grep ;)
[18:14:22] <jeremyb>	 die may be inaccurate :)
[18:14:32] <jeremyb>	 but i got it
[18:14:57] <Nemo_bis>	 or the disk, rather
[18:16:04] <awight>	 rats.  Looks like Nemo_bis is right that wfDebug > /dev/null
[18:17:26] <Nemo_bis>	 awight: what's the size of all those logs currently, btw?
[18:17:58] <awight>	 mmm
[18:18:07] <awight>	 they are rotated daily, i believe
[18:18:33] <awight>	 seems that today's logs add up to 41GB
[18:18:54] <awight>	 and the archive is 506GB (so perhaps rotation is less frequent?)
[18:19:08] <awight>	 get a usb key ready ;)
[18:20:05] <Nemo_bis>	 thanks
[18:21:04] <Thehelpfulone>	 Nemo_bis, I thought it was just an intermittent issue with Chrome?
[18:21:20] <Nemo_bis>	 never heard of this
[18:21:30] <Nemo_bis>	 though yes, Chrome is horrible ;) and yes, he uses it
[18:23:00] <Thehelpfulone>	 yeah generally Mailman and Chrome have issues from time to time
[18:23:13] <Thehelpfulone>	 I've had problems with time outs etc too, even when trying to modify list settings
[18:24:00] <Thehelpfulone>	 I think Mailman said words to the effect of "don't use Chrome" 
[18:25:44] <Nemo_bis>	 I see
[18:26:19] <jeremyb>	 what's wrong with the SMTP interface?
[18:26:39] <jeremyb>	 blank mail to ${listname}-unsubscribe@lists.wikimedia.org
[18:34:07] <Nemo_bis>	 wondering what this means https://ganglia.wikimedia.org/latest/graph.php?h=fluorine.eqiad.wmnet&m=cpu_report&r=year&s=by%20name&hc=4&mc=2&st=1384022007&g=network_report&z=medium&c=Miscellaneous%20eqiad
[20:57:00] <icinga-wm>	 PROBLEM - MySQL Slave Running on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[20:57:10] <icinga-wm>	 PROBLEM - MySQL Recent Restart on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[20:57:50] <icinga-wm>	 RECOVERY - MySQL Slave Running on db1047 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error:  
[20:58:00] <icinga-wm>	 RECOVERY - MySQL Recent Restart on db1047 is OK: OK 2698463 seconds since restart  
[22:35:59] <YuviPanda>	 MaxSem: did the UW disabling expensive queries patch get merged/
[22:36:09] <YuviPanda>	 ?
[22:36:45] <MaxSem>	 nope, because we didn't want to change the situation when there was higher-priority stuff to investigate
[22:37:18] <MaxSem>	 but it should - in graphite it ranges somewhere between 10 and 100 seconds
[22:37:33] <YuviPanda>	 MaxSem: *seconds*!?
[22:37:34] <YuviPanda>	 wat
[22:38:08] <MaxSem>	 https://graphite.wikimedia.org/dashboard/temporary-35
[22:39:16] <YuviPanda>	 MaxSem: i see some really tiny dots. what are the units for the Y axis?
[22:39:35] <MaxSem>	 ms
[22:39:57] <YuviPanda>	 ugh, ok
[22:47:02] <MaxSem>	 home sweet home, YuviPanda? :)
[22:47:13] <YuviPanda>	 MaxSem: don't get me started :P
[22:57:25] <ori-l>	 hey YuviPanda
[22:57:34] <YuviPanda>	 hi ori-l :D
[22:57:34] <ori-l>	 glad you made it home safe
[22:57:53] <YuviPanda>	 ori-l: yeah, and thanks to a long sandwich before the flight didn't have to eat shitty flight food :D
[22:58:58] <ori-l>	 heh
[23:06:03] <YuviPanda>	 MaxSem: feel free to disable UW's expensive stuff if that's causing trouble, I can find a solution for it later on
[23:06:35] <MaxSem>	 will do on Tue as part of deployment training
[23:06:45] <YuviPanda>	 MaxSem: ah, okay
[23:06:49] <YuviPanda>	 MaxSem: can you merge https://gerrit.wikimedia.org/r/#/c/94586/?
[23:06:53] <YuviPanda>	 trivial trivial trivial
[23:06:57] <YuviPanda>	 i would merge it myself but COI
[23:07:00] <YuviPanda>	 :P
[23:07:56] <MaxSem>	 done
[23:08:03] <YuviPanda>	 ty MaxSem
[23:08:07] <ori-l>	 MaxSem: whom are you training?
[23:08:35] <YuviPanda>	 MaxSem: maybe I should join in on the training too, even though I don't have deploy rights
[23:08:55] <MaxSem>	 it's for those who don't have em
[23:09:17] <ori-l>	 MaxSem: can I recommend http://asciinema.org/ ?
[23:09:26] <ori-l>	 you can record your terminal session really easily using that tool
[23:09:32] <YuviPanda>	 MaxSem: aha!
[23:09:45] <ori-l>	 you can install it locally and then ssh
[23:09:57] <MaxSem>	 we will do it via hangouts
[23:10:27] <ori-l>	 are you going to record it?
[23:10:40] <ori-l>	 the reason I'm asking is twofold -- one, there are folks like bd808 that need some training too
[23:10:54] <MaxSem>	 ohai bd808
[23:11:02] <ori-l>	 two, it'd be useful to hear you describe what works and what sucks about the process
[23:11:14] <ori-l>	 since core platform & ops are working on improving it
[23:11:36] <MaxSem>	 if it's gonna be git-deploy, what's the matter?
[23:12:14] <ori-l>	 there's a lot of the deployment process that git-deploy will simply invoke via hooks
[23:12:20] * bd808  waves his untrained n00b hand at MaxSem 

[23:12:27] <ori-l>	 but how that stuff works and what it does and what logs it emits, etc hasn't been determined yet
[23:13:24] <MaxSem>	 bd808, wanna join us?
[23:14:06] <bd808>	 MaxSem: Sure.
[23:15:45] <bd808>	 MaxSem: What time on Tuesday?
[23:16:31] <MaxSem>	 2-4pm, see your inbox
[23:17:11] <MaxSem>	 now for realz
[23:17:12] <ori-l>	 MaxSem: yay thanks!
[23:17:25] * bd808  waits for gmail which is sloooow

[23:17:50] <MaxSem>	 it's me who's a gcal noob
[23:21:56] <ori-l>	 l33t gcal skillx0rs
[23:22:55] <bd808>	 MaxSem: Got it. Overlaps with Core weekly but watching deploy stuff sounds more entertaining
[23:43:39] <icinga-wm>	 PROBLEM - Puppet freshness on sq48 is CRITICAL: No successful Puppet run in the last 10 hours