[00:13:08] <icinga-wm>	 PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours  
[00:33:58] <icinga-wm>	 RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Mon Sep 30 00:33:55 UTC 2013  
[00:34:08] <icinga-wm>	 PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours  
[00:49:15] <icinga-wm>	 PROBLEM - Puppet freshness on sockpuppet is CRITICAL: No successful Puppet run in the last 10 hours  
[01:03:06] <grrrit-wm>	 (03PS1) 10Springle: repool db1049 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86631 
[01:03:33] <grrrit-wm>	 (03CR) 10Springle: [C: 032] repool db1049 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86631 (owner: 10Springle)
[01:03:35] <icinga-wm>	 RECOVERY - search indices - check lucene status page on search1018 is OK: HTTP OK: HTTP/1.1 200 OK - 60696 bytes in 0.010 second response time  
[01:04:23] <logmsgbot>	 !log springle synchronized wmf-config/db-eqiad.php  'repool db1049'
[01:04:40] <morebots>	 Logged the message, Master
[01:08:46] <icinga-wm>	 PROBLEM - Puppet freshness on db1049 is CRITICAL: No successful Puppet run in the last 10 hours  
[01:08:56] <icinga-wm>	 PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours  
[01:09:46] <icinga-wm>	 PROBLEM - Puppet freshness on db1049 is CRITICAL: No successful Puppet run in the last 10 hours  
[01:10:46] <icinga-wm>	 PROBLEM - Puppet freshness on db1049 is CRITICAL: No successful Puppet run in the last 10 hours  
[01:11:47] <icinga-wm>	 PROBLEM - Puppet freshness on db1049 is CRITICAL: No successful Puppet run in the last 10 hours  
[01:12:46] <icinga-wm>	 PROBLEM - Puppet freshness on db1049 is CRITICAL: No successful Puppet run in the last 10 hours  
[01:13:46] <icinga-wm>	 PROBLEM - Puppet freshness on db1049 is CRITICAL: No successful Puppet run in the last 10 hours  
[01:14:46] <icinga-wm>	 PROBLEM - Puppet freshness on db1049 is CRITICAL: No successful Puppet run in the last 10 hours  
[01:15:46] <icinga-wm>	 PROBLEM - Puppet freshness on db1049 is CRITICAL: No successful Puppet run in the last 10 hours  
[01:16:46] <icinga-wm>	 PROBLEM - Puppet freshness on db1049 is CRITICAL: No successful Puppet run in the last 10 hours  
[01:17:46] <icinga-wm>	 PROBLEM - Puppet freshness on db1049 is CRITICAL: No successful Puppet run in the last 10 hours  
[01:18:46] <icinga-wm>	 PROBLEM - Puppet freshness on db1049 is CRITICAL: No successful Puppet run in the last 10 hours  
[01:19:46] <icinga-wm>	 PROBLEM - Puppet freshness on db1049 is CRITICAL: No successful Puppet run in the last 10 hours  
[01:25:46] <icinga-wm>	 PROBLEM - Puppet freshness on db1049 is CRITICAL: No successful Puppet run in the last 10 hours  
[01:33:56] <icinga-wm>	 RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Mon Sep 30 01:33:53 UTC 2013  
[01:34:56] <icinga-wm>	 PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours  
[01:48:58] <springle>	 !log upgrading db1038 to precise + mariadb
[01:49:08] <morebots>	 Logged the message, Master
[02:06:15] <grrrit-wm>	 (03PS1) 10Springle: reassign db1038 to s3 [operations/puppet] - 10https://gerrit.wikimedia.org/r/86633 
[02:07:44] <grrrit-wm>	 (03CR) 10Springle: [C: 032] reassign db1038 to s3 [operations/puppet] - 10https://gerrit.wikimedia.org/r/86633 (owner: 10Springle)
[02:15:09] <logmsgbot>	 !log LocalisationUpdate completed (1.22wmf18) at Mon Sep 30 02:15:09 UTC 2013
[02:15:25] <morebots>	 Logged the message, Master
[02:27:04] <logmsgbot>	 !log LocalisationUpdate completed (1.22wmf19) at Mon Sep 30 02:27:03 UTC 2013
[02:27:17] <morebots>	 Logged the message, Master
[05:11:32] <icinga-wm>	 PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours  
[08:07:36] <grrrit-wm>	 (03CR) 10Akosiaris: [C: 032] EventLogging on vanadium: move MongoDB data from /srv to /a [operations/puppet] - 10https://gerrit.wikimedia.org/r/86638 (owner: 10Ori.livneh)
[08:07:54] <paravoid>	 noo
[08:08:01] <paravoid>	 I think we decided to not do it
[08:08:07] <ori-l>	 yeah, sorry
[08:08:16] <ori-l>	 i should have abandoned it, was distracted
[08:08:16] <akosiaris>	 arg
[08:08:19] <ori-l>	 my bad :/
[08:08:30] <akosiaris>	 ok... resetting HEAD^
[08:08:49] <grrrit-wm>	 (03PS1) 10Ori.livneh: Revert "EventLogging on vanadium: move MongoDB data from /srv to /a" [operations/puppet] - 10https://gerrit.wikimedia.org/r/86639 
[08:09:05] <ori-l>	 nah, just merge the revert on top, i think
[08:09:08] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: php_admin_flag on for bits [operations/apache-config] - 10https://gerrit.wikimedia.org/r/86640 
[08:10:32] <akosiaris>	 why did we decide not to do it ?
[08:12:36] <ori-l>	 akosiaris: because /srv is the proper location; /a is not a standard. faidon suggested remounting /a as /srv, which works for me
[08:13:50] <akosiaris>	 huh... the day I will not longer see /a will be a joyous one
[08:14:42] <hashar>	 paravoid: thank you for the memcached memory limit tweak on beta :-]
[08:33:56] <icinga-wm>	 RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Mon Sep 30 08:33:47 UTC 2013  
[08:47:17] <grrrit-wm>	 (03PS1) 10Danny B.: skwikisource: Disable upload [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86643 
[09:02:11] <icinga-wm>	 PROBLEM - Puppet freshness on ms-be1008 is CRITICAL: No successful Puppet run in the last 10 hours  
[09:03:11] <icinga-wm>	 PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: No successful Puppet run in the last 10 hours  
[09:05:11] <icinga-wm>	 PROBLEM - Puppet freshness on ms-be1011 is CRITICAL: No successful Puppet run in the last 10 hours  
[09:06:31] <icinga-wm>	 PROBLEM - etherpad_lite_process_running on zirconium is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/bin/sh /usr/share/etherpad-lite/bin/safeRun.sh  
[09:35:33] <icinga-wm>	 RECOVERY - etherpad_lite_process_running on zirconium is OK: PROCS OK: 2 processes with regex args ^/bin/sh /usr/share/etherpad-lite/bin/safeRun.sh  
[09:36:13] <icinga-wm>	 RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Mon Sep 30 09:36:03 UTC 2013  
[09:36:14] <icinga-wm>	 PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours  
[09:36:43] <icinga-wm>	 RECOVERY - Check status of defined EventLogging jobs on vanadium is OK: OK: All defined EventLogging jobs are runnning.  
[09:38:12] <grrrit-wm>	 (03PS2) 10Hashar: Change ptwiktionary logo [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85516 (owner: 10TTO)
[09:39:47] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Change ptwiktionary logo [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85516 (owner: 10TTO)
[09:39:55] <grrrit-wm>	 (03Merged) 10jenkins-bot: Change ptwiktionary logo [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/85516 (owner: 10TTO)
[09:41:14] <grrrit-wm>	 (03PS1) 10Akosiaris: Typo fix for etherpad-lite settings [operations/puppet] - 10https://gerrit.wikimedia.org/r/86648 
[09:41:21] <logmsgbot>	 !log hashar synchronized wmf-config/InitialiseSettings.php  'Point ptwiktionary logo to commons {{gerrit|85516}} {{bug|53685}}'
[09:41:35] <morebots>	 Logged the message, Master
[09:43:42] <grrrit-wm>	 (03CR) 10Akosiaris: [C: 032] Typo fix for etherpad-lite settings [operations/puppet] - 10https://gerrit.wikimedia.org/r/86648 (owner: 10Akosiaris)
[09:47:33] <icinga-wm>	 PROBLEM - etherpad_lite_process_running on zirconium is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/bin/sh /usr/share/etherpad-lite/bin/safeRun.sh  
[09:50:34] <icinga-wm>	 RECOVERY - etherpad_lite_process_running on zirconium is OK: PROCS OK: 2 processes with regex args ^/bin/sh /usr/share/etherpad-lite/bin/safeRun.sh  
[09:51:18] <grrrit-wm>	 (03Abandoned) 10Faidon Liambotis: Enable PHP on w/404.php on bits [operations/apache-config] - 10https://gerrit.wikimedia.org/r/85177 (owner: 10Reedy)
[09:52:30] <grrrit-wm>	 (03CR) 10Mark Bergsma: "(2 comments)" [operations/dns] - 10https://gerrit.wikimedia.org/r/86272 (owner: 10Ottomata)
[09:52:41] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 04-1] Adding entries for bits-lb.ulsfo, mobile-lb.ulsfo, and upload-lb.ulsfo [operations/dns] - 10https://gerrit.wikimedia.org/r/86272 (owner: 10Ottomata)
[10:05:13] <icinga-wm>	 PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours  
[10:05:58] <grrrit-wm>	 (03CR) 10Mark Bergsma: "These actually need AAAA records too. But, this is a good start. :)" [operations/dns] - 10https://gerrit.wikimedia.org/r/84098 (owner: 10Hashar)
[10:06:17] <logmsgbot>	 !log springle synchronized wmf-config/AdminSettings.php  'revert admin account. errors'
[10:06:21] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 032] "These actually need AAAA records too. But, this is a good start. :)" [operations/dns] - 10https://gerrit.wikimedia.org/r/84098 (owner: 10Hashar)
[10:06:29] <morebots>	 Logged the message, Master
[10:14:33] <logmsgbot>	 !log springle synchronized wmf-config/AdminSettings.php  'switch admin account, attempt #2'
[10:14:47] <morebots>	 Logged the message, Master
[11:57:06] <akosiaris>	 :-)
[12:07:04] <hashar>	 I still have to find out how to inject packages in my cow builder install :/
[12:07:32] <hashar>	 I attempted to mount my local dir in the cow builder instance and use scripting hooks  to alter apt.conf but that never worked..
[12:07:50] <mark>	 ori-l: nice work :)
[12:09:08] <mark>	 ideally we'd have some server side aggregation of client time stats, automatically aggregated into derived subnets/prefixes
[12:41:31] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Logo configuration for pt.wiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86651 (owner: 10Dereckson)
[12:41:40] <grrrit-wm>	 (03Merged) 10jenkins-bot: Logo configuration for pt.wiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86651 (owner: 10Dereckson)
[12:41:53] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Logo configuration for *.wiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86653 (owner: 10Dereckson)
[12:42:01] <grrrit-wm>	 (03Merged) 10jenkins-bot: Logo configuration for *.wiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86653 (owner: 10Dereckson)
[12:43:53] <logmsgbot>	 !log hashar synchronized wmf-config/InitialiseSettings.php  'wiktionary logos form 135px to 128px so they fit in the 160px tall element.  Ping Nemo_bis and Dereckson.'
[12:44:08] <morebots>	 Logged the message, Master
[12:44:27] <grrrit-wm>	 (03CR) 10Hashar: "deployed in production" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86653 (owner: 10Dereckson)
[12:45:06] <grrrit-wm>	 (03CR) 10Hashar: "deployed :)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86651 (owner: 10Dereckson)
[12:48:22] <grrrit-wm>	 (03CR) 10Dereckson: "For reference, the 128px width allow to restrict the height to 155px instead of 164px, and then to allow the logos to fit in the 160px hei" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86653 (owner: 10Dereckson)
[13:11:39] <icinga-wm>	 PROBLEM - Host ms-be1001 is DOWN: PING CRITICAL - Packet loss = 100%  
[13:43:29] <icinga-wm>	 RECOVERY - Host ms-be1001 is UP: PING OK - Packet loss = 0%, RTA = 0.33 ms  
[13:45:19] <grrrit-wm>	 (03PS1) 10Dzahn: add ironholds to role::analytics::users (RT #5831) [operations/puppet] - 10https://gerrit.wikimedia.org/r/86666 
[13:45:29] <icinga-wm>	 PROBLEM - swift-account-auditor on ms-be1001 is CRITICAL: Connection refused by host  
[13:45:39] <icinga-wm>	 PROBLEM - swift-object-updater on ms-be1001 is CRITICAL: Connection refused by host  
[13:45:39] <icinga-wm>	 PROBLEM - DPKG on ms-be1001 is CRITICAL: Connection refused by host  
[13:45:39] <icinga-wm>	 PROBLEM - SSH on ms-be1001 is CRITICAL: Connection refused  
[13:45:39] <icinga-wm>	 PROBLEM - swift-object-auditor on ms-be1001 is CRITICAL: Connection refused by host  
[13:45:39] <icinga-wm>	 PROBLEM - swift-object-replicator on ms-be1001 is CRITICAL: Connection refused by host  
[13:45:40] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] add ironholds to role::analytics::users (RT #5831) [operations/puppet] - 10https://gerrit.wikimedia.org/r/86666 (owner: 10Dzahn)
[13:45:49] <icinga-wm>	 PROBLEM - swift-account-server on ms-be1001 is CRITICAL: Connection refused by host  
[14:54:43] <yurik>	 mark, good morning, any update on https://gerrit.wikimedia.org/r/#/c/86258/ ?
[14:55:56] <mark>	 clearly not eh ;)
[14:57:08] <mark>	 akosiaris: so nas1 is reporting problems with its shelves
[14:57:13] <mark>	 could you have a look at some point?
[14:58:07] <ottomata>	 mark, foundation-lb, mediawiki-lb: leave in or remove?
[15:05:46] <mark>	 i'd say long term there's no reason why this would need to be set by varnish?
[15:05:51] <mark>	 why not in the mediawiki config?
[15:07:05] <yurik>	 we need to set the x-force-esi header at first for -TEST, later - for several smaller carriers, then - half of them, then - globally to true.  Changing backend to support lists of known carriers is slightly more involved
[15:07:34] <mark>	 once it's globally true, that header is no longer needed
[15:07:39] <yurik>	 correct
[15:16:39] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] re-add mw1041 to apaches dsh group or it didn't get current apache-sync [operations/puppet] - 10https://gerrit.wikimedia.org/r/86670 (owner: 10Dzahn)
[16:19:39] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] "This looks good. In case you haven't realized, this repository was originally forked from <https://github.com/ripienaar/gdash/> (into <htt" [operations/software/gdash] - 10https://gerrit.wikimedia.org/r/86672 (owner: 10Krinkle)
[16:21:29] <grrrit-wm>	 (03CR) 10Krinkle: "I didn't realise at the time this was a fork. Makes sense." [operations/software/gdash] - 10https://gerrit.wikimedia.org/r/86672 (owner: 10Krinkle)
[16:21:45] <mark>	 ottomata: better now?
[16:21:56] <grrrit-wm>	 (03CR) 10Ori.livneh: [V: 032] layout: Use local copy of jquery and tablesorter [operations/software/gdash] - 10https://gerrit.wikimedia.org/r/86672 (owner: 10Krinkle)
[16:22:53] <ottomata>	 yeah looks much better
[16:25:32] <ori-l>	 Ah, I was simply going to offer to walk him through with gdash in case he hadn't done it before and wanted to try.
[16:25:43] <ori-l>	 walk him through it, even
[16:25:59] <RoanKattouw>	 Oh, nice
[16:26:28] <gwicke>	 ori-l: we have instructions for Parsoid at https://wikitech.wikimedia.org/wiki/Parsoid
[16:27:20] <ori-l>	 with an ascii art data flow diagram and everything! nice
[16:30:50] <grrrit-wm>	 (03CR) 10Ori.livneh: "Krinkle, would you like to try git-deploy and sync this change? If so, you can follow the directions at <https://wikitech.wikimedia.org/wi" [operations/software/gdash] - 10https://gerrit.wikimedia.org/r/86672 (owner: 10Krinkle)
[16:33:48] <grrrit-wm>	 (03CR) 10Krinkle: "OK" [operations/software/gdash] - 10https://gerrit.wikimedia.org/r/86672 (owner: 10Krinkle)
[16:34:13] <grrrit-wm>	 (03PS1) 10Ottomata: lvs400[13] and cp400[13] are ganglia data sources in ulsfo [operations/puppet] - 10https://gerrit.wikimedia.org/r/86684 
[17:24:48] <Ryan_Lane>	 this is the math stuff, rewritten in perl
[17:24:55] <paravoid>	 #5270
[17:24:59] <paravoid>	 (RT)
[17:25:29] <MaxSem>	 Q: Why rewrite stuff written in one horrible language in another horrible language?
[17:26:26] <Reedy>	 Progress
[17:27:45] <mutante>	 "Ironically, the Debian MediaWiki maintainers have packaged the Math
[17:27:48] <grrrit-wm>	 (03PS1) 10Ottomata: Puppetizing lvs400[1-4] in ulsfo. [operations/puppet] - 10https://gerrit.wikimedia.org/r/86690 
[17:27:48] <mutante>	 source) package"
[17:27:51] <mutante>	 extension and have already split texvc into a separate binary (but not
[17:28:01] <paravoid>	 that sounds familiar
[17:28:29] <mutante>	 heh, yea, you said that
[17:28:42] <grrrit-wm>	 (03CR) 10Ottomata: [C: 04-1] "I'm unsure about the bottom of the lvs4* node block. Do these need the IP address aliases?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/86690 (owner: 10Ottomata)
[17:28:46] <mutante>	 and then there was a suggestion by Reedy to split it off into a separate repo
[17:29:13] <mutante>	 "Since quantal, there is a package specifically for this,
[17:29:15] <mutante>	 http://packages.ubuntu.com/quantal/mediawiki-math-texvc"
[17:46:17] <Reedy>	 Fatal error:  Invalid host name (docroot=/usr/local/apache/common/docroot/bits/), can't determine language.#012 in /usr/local/apache/common-local/multiversion/M
[17:46:17] <Reedy>	 WMultiVersion.php on line 351
[17:46:42] <icinga-wm>	 RECOVERY - NTP on ms-be1001 is OK: NTP OK: Offset -0.0392717123 secs  
[17:47:11] <Ryan_Lane>	 is there any major benefit we're getting from switching away from the ocaml based rendering we're currently doing by switching to this perl based service?
[17:47:29] <Reedy>	 bugs!
[17:47:44] <Ryan_Lane>	 it seems like it's going to be just as hard to deploy and maintain
[17:47:47] <Ryan_Lane>	 maybe harder
[17:47:53] <AaronSchulz>	 some bugs taste better than others
[17:48:11] <Ryan_Lane>	 I can see a major benefit to just doing it in nodejs + mathjax
[17:49:13] <physikerwelt>	 ok there is no ocaml based rendering now. The rendering is done by latex. LaTeXML does the conversion from tex to mathml only.
[17:49:34] <Ryan_Lane>	 I know. but we currently render using texvc
[17:49:43] <Ryan_Lane>	 it works, it just isn't maintained
[17:49:49] <Reedy>	 paravoid: Can you look what PHP things seem to be running on bits and failing and fatalling in MWMultiVersion?
[17:49:50] <Ryan_Lane>	 it's relatively easy to deploy
[17:50:06] <Ryan_Lane>	 what's being proposed is more complicated and is harder to deploy
[17:50:10] <physikerwelt>	 and texvc calls latex
[17:50:17] <paravoid>	 Reedy: what do you mean?
[17:50:21] <Ryan_Lane>	 I've hard talk of using nodejs + mathjax
[17:50:34] <Ryan_Lane>	 and that is simpler, easier to maintain, and easier to deploy
[17:50:39] <Reedy>	 Sep 30 17:46:04 10.64.16.131 apache2[5072]: PHP Fatal error:  Invalid host name (docroot=/usr/local/apache/common/docroot/bits/), can't determine language.#012 in /usr/local/apache/common-local/multiversion/MWMultiVersion.php on line 351
[17:50:52] <Ryan_Lane>	 it also uses the same code on the client and the server
[17:50:55] <Reedy>	 The fatal is caused by our code
[17:51:03] <Reedy>	 I'm just slightly confused what's apparently trying to be run on bits
[17:51:32] <Ryan_Lane>	 it just seems like we're replacing one unmaintained service with another that will be harder to deploy
[17:51:46] <physikerwelt>	 mathjax and nodejs is not ready now. latexml is
[17:52:02] <icinga-wm>	 PROBLEM - MySQL Processlist on db1052 is CRITICAL: CRIT 1 unauthenticated, 0 locked, 0 copy to table, 72 statistics  
[17:53:02] <icinga-wm>	 PROBLEM - MySQL Processlist on db1051 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 79 statistics  
[17:53:10] <Ryan_Lane>	 physikerwelt: who's the project owner for this on the WMF side?
[17:53:19] <physikerwelt>	 I have somehow the feeling to be stuck in an infitiv loop.
[17:53:28] <physikerwelt>	 I have talked to gabriel
[17:53:37] <Ryan_Lane>	 well, that's because I think a lot of people are kind of iffy about this solution
[17:54:08] <Ryan_Lane>	 we'd be rushing to replace something that's currently working with something new that's not an ideal solution
[17:54:20] <grrrit-wm>	 (03CR) 10Dereckson: "Logos are now protected." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86660 (owner: 10Dereckson)
[17:54:59] <Ryan_Lane>	 people generally don't like replacing something they don't like with something they like only slightly more ;)
[17:55:02] <icinga-wm>	 RECOVERY - MySQL Processlist on db1051 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 4 statistics  
[17:55:02] <icinga-wm>	 RECOVERY - MySQL Processlist on db1052 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 2 statistics  
[17:55:06] <physikerwelt>	 the new solution is oackwards compatible
[17:55:13] <MaxSem>	 how?
[17:55:25] <Ryan_Lane>	 that isn't necessarily the concern
[17:55:33] <Ryan_Lane>	 the concern is: what does this gain us over the old solution?
[17:55:41] <physikerwelt>	 mathml output
[17:55:47] <Ryan_Lane>	 and: does it gain us enough to be worth the effort?
[17:56:06] <Ryan_Lane>	 and: if there's a better solution on the horizon, is it worth waiting for it?
[17:56:10] <physikerwelt>	 I have discussed that before travelling to sfo and the answer was yes
[17:56:33] <Ryan_Lane>	 how long till mathjax + nodejs is possible?
[17:56:59] <physikerwelt>	 They said something like november
[17:57:31] <Ryan_Lane>	 I think it's likely worth waiting two months
[17:58:02] <icinga-wm>	 PROBLEM - MySQL Processlist on db1052 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 93 statistics  
[17:58:07] <Ryan_Lane>	 it could take two months for this to get deployed
[17:58:38] <physikerwelt>	 I think that's somehow unfair
[17:58:45] <MaxSem>	 and think of all the kittens...
[17:58:53] <Ryan_Lane>	 it needs to have a security audit of the code, it needs to have hardware allocated, it needs ops support to deploy it
[17:59:02] <icinga-wm>	 PROBLEM - MySQL Processlist on db1051 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 69 statistics  
[17:59:02] <icinga-wm>	 RECOVERY - MySQL Processlist on db1052 is OK: OK 1 unauthenticated, 0 locked, 0 copy to table, 6 statistics  
[17:59:07] <Ryan_Lane>	 physikerwelt: all new projects take ages
[17:59:44] <YuviPanda>	 Ryan_Lane: is there any link with notes on this 'mathjax + nodejs' you mentioned?
[17:59:56] <Ryan_Lane>	 it was a thread on the ops list
[18:00:04] <YuviPanda>	 oh, ok
[18:01:02] <icinga-wm>	 RECOVERY - MySQL Processlist on db1051 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 2 statistics  
[18:01:04] <physikerwelt>	 I really don't undstand that.
[18:01:19] <Ryan_Lane>	 physikerwelt: all I'm saying is that it's likely better to spend effort on the solution that we really want, especially if it's going to be ready in a couple months
[18:01:41] <YuviPanda>	 Ryan_Lane: only thing I find is the thread of 'shelling out vs new service'
[18:01:44] <YuviPanda>	 Ryan_Lane: nothing about nodejs
[18:02:13] <physikerwelt>	 I doubt that it will be ready in a couple of month
[18:02:14] <YuviPanda>	 Ryan_Lane: and IIRC, 'new service', the option that got a good bit of support on that thread, is what physikerwelt is now proposing?
[18:02:22] <Ryan_Lane>	 "In the longer term, the plan is to go for a web service that also generates SVG, likely using node.js and MathJax"
[18:02:25] <grrrit-wm>	 (03CR) 10Dereckson: "@Adamw What's the best practice to include a MediaWiki core file in CommonSettings.php/throttle.php/InitialiseSettings.php?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/65644 (owner: 10Dereckson)
[18:02:48] <physikerwelt>	 latexml service works since more than a year stable and open to the public in the web
[18:03:23] <Ryan_Lane>	 I'm not saying no.
[18:03:34] <physikerwelt>	 there are quite a few options to convert tex to mathml after literature research latexml was considerd as the best
[18:03:38] <Ryan_Lane>	 but I don't think we actually have anyone assigned to this project
[18:03:47] <Ryan_Lane>	 which means getting someone to set aside time to do this
[18:03:55] <YuviPanda>	 Ryan_Lane: so, from my understanding (I looked at the new math stuff a bit y'day) - node.js and MathJax can only produce SVG, while we need PNGs for backwards compat anyway. So we could maybe have an *additional* thing that's nodejs + MathJax, but in the end we still need somthing that puts out PNG
[18:04:30] <Ryan_Lane>	 YuviPanda: can output svg and also convert svg to png
[18:04:39] <Ryan_Lane>	 not an ideal solution, of course
[18:05:02] <icinga-wm>	 PROBLEM - MySQL Processlist on db1051 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 70 statistics  
[18:05:02] <icinga-wm>	 PROBLEM - MySQL Processlist on db1052 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 106 statistics  
[18:05:12] <Ryan_Lane>	 anyway, we can deploy the code using git-deploy
[18:05:18] <Ryan_Lane>	 and we can have a module that runs the make
[18:05:25] <Ryan_Lane>	 assuming the code is security audited
[18:05:38] <Ryan_Lane>	 and someone sets aside time to do the ops work
[18:05:53] <Ryan_Lane>	 I just thought I'd raise the concerns ;)
[18:06:40] <physikerwelt>	 ok that said there is no chance to get latexml deployed and I can fly home
[18:23:22] <grrrit-wm>	 (03PS3) 10Andrew Bogott: Move maven package def into contint module. [operations/puppet] - 10https://gerrit.wikimedia.org/r/86136 
[18:23:24] <paravoid>	 bd808: I haven't forgot about you, but it'll take a day or two more for me to take a careful look
[18:23:36] <paravoid>	 bd808: sorry for the delay
[18:23:50] <bd808>	 paravoid: No worries. THanks for taking the time to help.
[18:24:28] <grrrit-wm>	 (03PS1) 10Reedy: Everything non wikipedia to 1.22wmf19 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86700 
[18:24:47] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Everything non wikipedia to 1.22wmf19 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86700 (owner: 10Reedy)
[18:24:55] <grrrit-wm>	 (03Merged) 10jenkins-bot: Everything non wikipedia to 1.22wmf19 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86700 (owner: 10Reedy)
[18:25:35] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Move maven package def into contint module. [operations/puppet] - 10https://gerrit.wikimedia.org/r/86136 (owner: 10Andrew Bogott)
[18:27:29] <mutante>	 physikerwelt: could you post a proposal on wikitech-l? that would be best for ongoing discussion because we are all in a meeting now
[18:28:27] <Ryan_Lane>	 physikerwelt: that was our consensus from the meeting, btw
[18:28:28] <yurik>	 bblack, around?
[18:28:34] <Ryan_Lane>	 is that this should be brought up on wikitech-l
[18:32:42] <physikerwelt>	 For me it seems to be an infitiv look... http://wikimedia.7.x6.nabble.com/Display-of-lt-math-gt-using-LaTeXML-td4988325.html
[18:33:12] <grrrit-wm>	 (03CR) 10Dereckson: "Follow-up: I6806c875e4d16102d4a7c106e2fd0fe4edf36e45" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86653 (owner: 10Dereckson)
[18:33:42] <physikerwelt>	 http://arxiv.org/abs/1304.5475
[18:33:57] <icinga-wm>	 RECOVERY - Puppet freshness on labstore4 is OK: puppet ran at Mon Sep 30 18:33:49 UTC 2013  
[18:34:47] <icinga-wm>	 PROBLEM - Puppet freshness on labstore4 is CRITICAL: No successful Puppet run in the last 10 hours  
[18:35:36] <bblack>	 yurik: yes
[18:39:19] <andrewbogott>	 physikerwelt: It looks like you've done a good job of explaining why this feature is a good idea -- now we need a discussion about how it should be implemented.  Those decisions will involve people from multiple teams (not just Ops), and wikitech-l is the best way to invite all those people to the discussion.
[18:41:51] <physikerwelt>	 I don't understand what needs to be implemented... I have no abition to keep posting to wikitech-l without to get any feedback
[18:42:18] <grrrit-wm>	 (03PS1) 10Yurik: Set X-CS for all carriers identified by netmapper by default [operations/puppet] - 10https://gerrit.wikimedia.org/r/86708