[00:40:18] <icinga-wm>	 PROBLEM - DPKG on osmium is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[00:41:27] <icinga-wm>	 RECOVERY - DPKG on osmium is OK: All packages OK  
[02:22:04] <logmsgbot>	 !log LocalisationUpdate completed (1.24wmf11) at 2014-06-29 02:21:01+00:00
[02:22:10] <morebots>	 Logged the message, Master
[02:45:41] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Jun 29 02:44:35 UTC 2014 (duration 44m 34s)
[02:45:45] <morebots>	 Logged the message, Master
[04:17:54] <icinga-wm>	 PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Sun 29 Jun 2014 02:16:53 UTC  
[04:57:21] <icinga-wm>	 RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Sun Jun 29 04:57:15 UTC 2014  
[06:07:05] <Bsadowski1>	 I'm having issues with CSS.
[06:58:58] <grrrit-wm>	 (03PS3) 10Yuvipanda: Add Grafana module & role [operations/puppet] - 10https://gerrit.wikimedia.org/r/133274 (owner: 10Ori.livneh)
[06:59:34] <grrrit-wm>	 (03CR) 10Yuvipanda: "Manual rebase + realm branching in the role to point to the labs URL." [operations/puppet] - 10https://gerrit.wikimedia.org/r/133274 (owner: 10Ori.livneh)
[07:23:07] <grrrit-wm>	 (03CR) 10Yuvipanda: "Also why not just use nginx? (I can switch it over if others are ok with it)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/133274 (owner: 10Ori.livneh)
[07:37:32] <grrrit-wm>	 (03PS4) 10Yuvipanda: Add Grafana module & role [operations/puppet] - 10https://gerrit.wikimedia.org/r/133274 (owner: 10Ori.livneh)
[07:39:23] <grrrit-wm>	 (03PS5) 10Yuvipanda: Add Grafana module & role [operations/puppet] - 10https://gerrit.wikimedia.org/r/133274 (owner: 10Ori.livneh)
[07:44:59] <grrrit-wm>	 (03PS6) 10Yuvipanda: Add Grafana module & role [operations/puppet] - 10https://gerrit.wikimedia.org/r/133274 (owner: 10Ori.livneh)
[07:58:34] <icinga-wm>	 PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Sun 29 Jun 2014 05:57:45 UTC  
[08:43:00] <grrrit-wm>	 (03CR) 10Yuvipanda: "Switched over to nginx, and also realmed the monitoring." [operations/puppet] - 10https://gerrit.wikimedia.org/r/133274 (owner: 10Ori.livneh)
[08:55:03] <grrrit-wm>	 (03PS7) 10Yuvipanda: Add Grafana module & role [operations/puppet] - 10https://gerrit.wikimedia.org/r/133274 (owner: 10Ori.livneh)
[09:17:02] <icinga-wm>	 RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Sun Jun 29 09:16:58 UTC 2014  
[09:19:50] <grrrit-wm>	 (03CR) 10Yuvipanda: "So I've a version of grafana running on charcoal-test.wmflabs.org! \o/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/133274 (owner: 10Ori.livneh)
[09:33:05] <grrrit-wm>	 (03CR) 10Yuvipanda: "The version of Grafana in gerrit is also fairly outdated and buggy. I've updated the grafana on charcoal-test to be latest master, and tha" [operations/puppet] - 10https://gerrit.wikimedia.org/r/133274 (owner: 10Ori.livneh)
[12:13:04] <icinga-wm>	 PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sun 29 Jun 2014 10:12:53 UTC  
[12:39:25] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 04-2] "Haha, nice -- I like the list of reviewers especially :)" [operations/dns] - 10https://gerrit.wikimedia.org/r/142622 (owner: 10Dzahn)
[12:50:27] <matanya>	 paravoid: OTRS admins report no more Bayes-scoring on OTRS/iodine snice 3 days
[12:50:33] <matanya>	 I guess it is related to https://gerrit.wikimedia.org/r/#/c/141919/  
[12:51:10] <Reedy>	 matanya: I dumped the irc log onto the ticket ;)
[12:51:56] <matanya>	 sweet Reedy thanks
[12:52:20] <Reedy>	 I presumed it'd at least make its way to his inbox, rather than relying on pings and backscroll
[12:52:34] <matanya>	 makes sense
[12:52:55] <matanya>	 since it was around 1am for me sense already went to sleep
[13:46:04] <icinga-wm>	 PROBLEM - puppet last run on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[13:46:54] <icinga-wm>	 RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 1178 seconds ago with 0 failures  
[13:53:15] <icinga-wm>	 RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sun Jun 29 13:53:03 UTC 2014  
[15:17:45] <icinga-wm>	 PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Sun 29 Jun 2014 13:17:11 UTC  
[15:36:41] <icinga-wm>	 RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Sun Jun 29 15:36:35 UTC 2014  
[16:02:33] <ori>	 YuviPanda|brb: we need apache for ldap
[16:44:30] <hoo>	 !log Jenkins/ Zuul not reacting for at least half an hour now
[16:44:35] <morebots>	 Logged the message, Master
[16:46:01] <hoo>	 ^d: Reedy: --^
[16:46:09] <hoo>	 ori: ^
[17:23:00] <grrrit-wm>	 (03CR) 10BryanDavis: "For elasticsearch integration, see the reverse proxy configuration example in the graphana upstream (https://github.com/grafana/grafana/bl" [operations/puppet] - 10https://gerrit.wikimedia.org/r/133274 (owner: 10Ori.livneh)
[18:01:26] <YuviPanda|brb>	 ori: heya
[18:02:11] <YuviPanda>	 ori: started physio for the hand today. only one hand usable for next 2 weeks
[18:03:36] <YuviPanda>	 ori: right. i'll move it back
[18:03:47] <YuviPanda>	 ori: but no auth for labs tho
[20:17:37] <icinga-wm>	 PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Sun 29 Jun 2014 18:17:22 UTC  
[20:23:13] <grrrit-wm>	 (03Abandoned) 10Dereckson: Add skipcaptcha right for all sysops on labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/115230 (owner: 10Dereckson)
[20:30:37] <YuviPanda>	 ori: hmm, do we monitor server response codes / response times on prod and push them to graphite? if so do you know what we use to do that?
[20:35:06] * YuviPanda goes to sleep again
[20:57:39] <icinga-wm>	 RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Sun Jun 29 20:57:37 UTC 2014  
[21:24:39] <bawolff>	 If anyone's around (and not busy), could someone look up what [5f513816] 2014-06-29 21:18:24 is in the exception log (for bug 60629)
[21:24:58] <hoo>	 bawolff: Can do
[21:25:15] <bawolff>	 yay :)
[21:26:30] <hoo>	 bawolff: http://fpaste.org/114217/40407717/
[21:26:48] <hoo>	 If you open a bug, please copy that in as the paste will expire in a month
[21:27:35] <bawolff>	 Verdy alread opened up a bug (Or should I say one person talking match)
[21:29:48] <bawolff>	 Ah translate extension. Always causing trouble ;)
[21:30:08] <hoo>	 Often, Yeah :/
[21:31:25] <bawolff>	 silly people wanting to speak their own language
[22:10:49] <hoo>	 !log Manually cleared a watchlist with 289436 entries, see https://bugzilla.wikimedia.org/show_bug.cgi?id=67123#c5
[22:10:54] <morebots>	 Logged the message, Master
[22:16:53] <Bsadowski1>	 200,000+ pages, Kolega2357?
[22:17:05] <Kolega2357>	 Bsadowski1 yes
[22:17:10] <Bsadowski1>	 lol
[22:17:12] <hoo>	 Yes... even Watchlist/clear was giving fatal errors for him
[22:17:13] <hoo>	 see the bug
[22:17:17] <Bsadowski1>	 I saw.
[22:17:43] <icinga-wm>	 PROBLEM - puppetmaster backend https on palladium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8141: HTTP/1.1 500 Internal Server Error  
[22:21:19] <Kolega2357>	 hoo https://bugzilla.wikimedia.org/show_bug.cgi?id=67123#c6
[22:26:11] <hoo>	 !log Manually cleared a watchlist on shwikt with 819846 entries, see https://bugzilla.wikimedia.org/show_bug.cgi?id=67123#c7
[22:26:15] <morebots>	 Logged the message, Master
[22:28:10] <Kolega2357>	 hoo Thanks
[22:28:53] <icinga-wm>	 PROBLEM - puppetmaster https on palladium is CRITICAL: HTTP CRITICAL - No data received from host  
[22:29:44] <icinga-wm>	 RECOVERY - puppetmaster backend https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.072 second response time  
[22:29:53] <icinga-wm>	 RECOVERY - puppetmaster https on palladium is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.013 second response time  
[22:37:11] <Kolega2357>	 Bsadowski1 Also cleared:
[22:37:11] <Kolega2357>	 hoo@terbium:~$ sudo -u apache mwscript runBatchedQuery.php --wiki shwiktionary 'DELETE FROM watchlist WHERE wl_user = 977 LIMIT 5000'
[22:37:11] <Kolega2357>	 (819846 entries)
[22:38:58] <Reedy>	 srsly
[22:44:09] <Bsadowski1>	 I saw. I'm not blind
[22:44:47] <Reedy>	 Bar an errant script adding pages to your watchlist...
[22:44:54] <Reedy>	 How the hell do you end up with anywhere near that many?
[22:47:25] <bawolff>	 Maybe if you have the auto-add things i edit to my watchlist pref set, and then run a bot from your account or something?
[22:51:50] <Reedy>	 That's what I was meaning by errant script I guess
[23:10:15] <Kolega2357>	 Why MediaWiki software on sh wikipedia not correcly sorting article?
[23:10:57] <Reedy>	 Depends what you mean by "correctly"
[23:11:11] <Kolega2357>	 Some letters is not good sorting
[23:11:22] <Reedy>	 Presuambly as it's not using the default 'uppercase' category collation
[23:11:45] <Kolega2357>	 How to set category collation?
[23:11:54] <bawolff>	 I don't think we have support for sh
[23:12:39] <Reedy>	 Doesn't look like it
[23:12:55] <Reedy>	 Kolega2357: You might want to open a couple of bugs
[23:12:57] <Reedy>	 There's 2 parts to it
[23:13:15] <Kolega2357>	 Reedy Can I am have support of community?
[23:13:39] <Reedy>	 And there's maybe some questions to be asked to a native speaker
[23:13:56] <bawolff>	 Kolega2357: Get sh added to the cldr collation list first
[23:14:00] <Reedy>	 The MediaWiki support side of things doesn't need it, but deploying on shwiki needs doing first
[23:14:03] <bawolff>	 That will take a really long time probably
[23:14:08] <MatmaRex>	 i wonder if 'hr' would be close enough?
[23:14:15] <Reedy>	 bawolff: oh, like, srsly upstream it's not supported?
[23:14:22] <Reedy>	 libicu etc?
[23:14:25] <bawolff>	 upstream seriously sucks for collations
[23:14:37] <bawolff>	 http://unicode.org/cldr/trac/browser/trunk/common/collation/ is the list
[23:15:03] <Reedy>	 lol
[23:15:09] <MatmaRex>	 they use the same alphabet apparently, https://en.wikipedia.org/wiki/Gaj's_Latin_alphabet
[23:16:09] <Kolega2357>	 MatmaRex yes you are a right Gaj's_Latin_alphabet is need for sh language
[23:16:25] <bawolff>	 Reedy: libicu has a cool feature where you can make custom collations. But the php bindings don't support it
[23:17:06] <MatmaRex>	 bawolff: we could probably just set uca-hr on sh.wp
[23:17:07] <Kolega2357>	 Reedy http://prntscr.com/3xt2ix What I am select part for bug?
[23:17:18] <MatmaRex>	 per what Kolega2357 is saying
[23:17:43] <bawolff>	 Kolega2357: Wikimedia and then Site Requests
[23:18:44] <Kolega2357>	 MatmaRex yes  uca-hr can for sh wiki. 
[23:18:52] <Kolega2357>	 bawolff I am wirte bug
[23:21:49] <Kolega2357>	 MatmaRex Can is possible category collation for sh wiktionary?
[23:22:33] <MatmaRex>	 maybe. i don't know
[23:23:23] <Kolega2357>	 https://bugzilla.wikimedia.org/show_bug.cgi?id=67287
[23:23:29] <Kolega2357>	 MatmaRex https://bugzilla.wikimedia.org/show_bug.cgi?id=67287
[23:24:05] <bawolff>	 Kolega2357: http://developer.mimer.com/charts/croatian.htm would be the new sort order. Does that look good for your language?
[23:24:50] <Kolega2357>	 bawolff Yes. No need new sort order.
[23:26:36] <bawolff>	 Kolega2357: Ok. You need to get support on wiki for the change
[23:27:38] <Kolega2357>	 bawolff I am afraid, several times i am accused for canvassing
[23:28:43] <bawolff>	 In order to make the change, we need to know that shwiki has been consulted
[23:32:23] <Kolega2357>	 bawolff How long time need for voting on sh wikipedia_
[23:33:05] <bawolff>	 A reasonable amount of time that active users see the vote
[23:33:20] <bawolff>	 Standards usually are set by the wiki community
[23:33:28] <bawolff>	 A week would probably be reasonable
[23:37:27] <Kolega2357>	 bawolff https://sh.wikipedia.org/wiki/Wikipedia:Pijaca-%D0%9F%D0%B8%D1%98%D0%B0%D1%86%D0%B0#.24wgCategoryCollation I am started voting