[00:21:17] <icinga-wm>	 PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Server Error - 1703 bytes in 8.207 second response time  
[00:26:35] <hoo|away>	 TimStarling: https://gerrit.wikimedia.org/r/74591 two things: that shouldn't run as root, also there's no /home/wikipedia on terbium
[00:26:44] <hoo|away>	 I can make a patch, just tell me where the logs should go
[00:31:19] <TimStarling>	 can you write a comment on the change in gerrit?
[00:31:24] <hoo|away>	 sure
[00:33:23] <grrrit-wm>	 (03CR) 10Hoo man: Properly puppeti[sz]e purge-checkuser (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/74591 (owner: 10Reedy)
[00:33:29] <hoo|away>	 there you go
[00:34:39] <grrrit-wm>	 (03PS1) 10Tim Starling: Revert "Properly puppeti[sz]e purge-checkuser" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122616 
[00:34:49] <grrrit-wm>	 (03CR) 10Tim Starling: [C: 032 V: 032] Revert "Properly puppeti[sz]e purge-checkuser" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122616 (owner: 10Tim Starling)
[00:35:03] <Bsadowski1>	 What does that do?
[00:35:19] <Bsadowski1>	 Or does it just help with the purging of the data?
[00:35:38] <hoo|away>	 Bsadowski1: It just puppetized the old way of doing things
[00:35:47] <hoo|away>	 nothing to worry about from an end user perpspective
[00:37:14] <hoo|away>	 TimStarling: Will you redo that one or shall I?
[00:37:25] <TimStarling>	 you can do it
[00:37:31] <hoo|away>	 ok, will do
[00:37:36] <TimStarling>	 I don't know why I'm involved in the first place
[00:38:04] <hoo|away>	 Sorry, thought it was your change :P
[00:38:11] <hoo|away>	 Damn gerrit mails -.-
[00:38:35] <hoo|away>	 From:	Tim Starling (Code Review) [...] Subject:	[Ops] [Gerrit] Properly puppeti[sz]e purge-checkuser - change (operations/puppet)
[00:38:55] <TimStarling>	 well, matanya asked me to merge it, so I clicked the button
[00:42:19] <TimStarling>	 Bsadowski1: normally, old checkuser rows are purged on one in every 1000 edits
[00:42:44] <TimStarling>	 that means you have an average of 500 edits older than the purge expiry time
[00:43:32] <TimStarling>	 on wikis that are read-only or rarely-edited, checkuser rows can persist for many years
[00:43:44] <TimStarling>	 which was a privacy policy compliance problem
[00:44:43] <TimStarling>	 so we introduced a cron job which would force a regular log purge
[00:46:55] <grrrit-wm>	 (03PS1) 10Hoo man: Puppetize purge-checkuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/122617 
[00:47:42] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Puppetize purge-checkuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/122617 (owner: 10Hoo man)
[00:48:44] <hoo|away>	 damn git ...
[00:49:01] <grrrit-wm>	 (03PS2) 10Hoo man: Puppetize purge-checkuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/122617 
[00:59:25] <logmsgbot>	 !log springle synchronized wmf-config/db-eqiad.php  's7 repool db1034, warm up'
[00:59:31] <morebots>	 Logged the message, Master
[01:15:38] <grrrit-wm>	 (03PS5) 10Springle: Fix the sql script to allow using the centralauth DB directly [operations/puppet] - 10https://gerrit.wikimedia.org/r/120730 (owner: 10Hoo man)
[01:18:46] <grrrit-wm>	 (03CR) 10Springle: [C: 032] "It would be nice if $sectionsByDB was used instead... somehow. But since the fawiki hack has been in for a while, let's make it a better " [operations/puppet] - 10https://gerrit.wikimedia.org/r/120730 (owner: 10Hoo man)
[01:24:21] <grrrit-wm>	 (03PS2) 10Springle: Add legalteamwiki to private_wikis [operations/puppet] - 10https://gerrit.wikimedia.org/r/122610 (owner: 10Reedy)
[01:24:32] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Add legalteamwiki to private_wikis [operations/puppet] - 10https://gerrit.wikimedia.org/r/122610 (owner: 10Reedy)
[01:27:35] <grrrit-wm>	 (03PS2) 10Springle: Added zerowiki to $private_wikis [operations/puppet] - 10https://gerrit.wikimedia.org/r/122588 (owner: 10Yurik)
[01:30:36] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Added zerowiki to $private_wikis [operations/puppet] - 10https://gerrit.wikimedia.org/r/122588 (owner: 10Yurik)
[01:34:59] <springle>	 !log restarted sanitarium s3 instance for additional private wikis
[01:35:08] <morebots>	 Logged the message, Master
[01:35:57] <grrrit-wm>	 (03PS1) 10Reedy: Grant reedy root [operations/puppet] - 10https://gerrit.wikimedia.org/r/122621 
[01:37:22] <grrrit-wm>	 (03PS1) 10Ori.livneh: Add report.py from noc.wikimedia.org/cgi-bin/report.py [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/122622 
[01:37:49] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032 V: 032] Add report.py from noc.wikimedia.org/cgi-bin/report.py [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/122622 (owner: 10Ori.livneh)
[01:39:08] <grrrit-wm>	 (03PS1) 10Ori.livneh: Revert "Add report.py from noc.wikimedia.org/cgi-bin/report.py" [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/122623 
[01:39:28] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032 V: 032] "Already moved to separate repo; I'm an idiot." [operations/software/mwprof] - 10https://gerrit.wikimedia.org/r/122623 (owner: 10Ori.livneh)
[01:40:51] <grrrit-wm>	 (03PS1) 10Ori.livneh: Make configurable; lint [operations/software/mwprof/reporter] - 10https://gerrit.wikimedia.org/r/122624 
[01:41:04] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032 V: 032] Make configurable; lint [operations/software/mwprof/reporter] - 10https://gerrit.wikimedia.org/r/122624 (owner: 10Ori.livneh)
[01:42:07] <icinga-wm>	 PROBLEM - Full LVS Snapshot on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[01:42:57] <icinga-wm>	 PROBLEM - RAID on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[01:42:57] <icinga-wm>	 RECOVERY - Full LVS Snapshot on db1047 is OK: OK no full LVM snapshot volumes  
[01:43:47] <icinga-wm>	 RECOVERY - RAID on db1047 is OK: OK: optimal, 3 logical, 6 physical  
[01:44:35] <grrrit-wm>	 (03CR) 10Springle: [C: 04-2] "Have found analytics data on db67. Mailing wmfresearch@ to find out if it's still important." [operations/puppet] - 10https://gerrit.wikimedia.org/r/122406 (owner: 10Dzahn)
[01:44:37] <grrrit-wm>	 (03CR) 10Springle: [C: 04-2] "Have found analytics data on db67. Mailing wmfresearch@ to find out if it's still important." [operations/dns] - 10https://gerrit.wikimedia.org/r/122412 (owner: 10Matanya)
[01:46:23] <grrrit-wm>	 (03PS1) 10Ori.livneh: add git deploy target for reporter [operations/puppet] - 10https://gerrit.wikimedia.org/r/122625 
[01:47:55] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] add git deploy target for reporter [operations/puppet] - 10https://gerrit.wikimedia.org/r/122625 (owner: 10Ori.livneh)
[01:49:40] <grrrit-wm>	 (03PS1) 10Ori.livneh: Amend location of report.cfg [operations/software/mwprof/reporter] - 10https://gerrit.wikimedia.org/r/122626 
[01:50:45] <grrrit-wm>	 (03PS2) 10Ori.livneh: Amend location of report.cfg [operations/software/mwprof/reporter] - 10https://gerrit.wikimedia.org/r/122626 
[01:51:03] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032 V: 032] Amend location of report.cfg [operations/software/mwprof/reporter] - 10https://gerrit.wikimedia.org/r/122626 (owner: 10Ori.livneh)
[01:55:57] <icinga-wm>	 PROBLEM - RAID on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[01:55:57] <icinga-wm>	 PROBLEM - MySQL Recent Restart on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[01:56:07] <icinga-wm>	 PROBLEM - MySQL Processlist on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[01:56:07] <icinga-wm>	 PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[01:56:12] * springle  sigh

[01:56:47] <icinga-wm>	 RECOVERY - MySQL Recent Restart on db1047 is OK: OK 528255 seconds since restart  
[01:56:47] <icinga-wm>	 RECOVERY - RAID on db1047 is OK: OK: optimal, 3 logical, 6 physical  
[01:56:57] <icinga-wm>	 RECOVERY - MySQL Processlist on db1047 is OK: OK 0 unauthenticated, 0 locked, 2 copy to table, 0 statistics  
[01:56:57] <icinga-wm>	 RECOVERY - MySQL Idle Transactions on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds  
[01:58:03] <springle>	 !log killed research queries on db1047. email me
[01:58:12] <morebots>	 Logged the message, Master
[02:02:42] <logmsgbot>	 !log LocalisationUpdate failed: git pull of extensions failed
[02:02:48] <morebots>	 Logged the message, Master
[02:09:19] <grrrit-wm>	 (03PS2) 10Springle: db: lint role [operations/puppet] - 10https://gerrit.wikimedia.org/r/122332 (owner: 10Matanya)
[02:10:54] <grrrit-wm>	 (03CR) 10Springle: [C: 032] db: lint role [operations/puppet] - 10https://gerrit.wikimedia.org/r/122332 (owner: 10Matanya)
[02:16:37] <grrrit-wm>	 (03PS1) 10Ori.livneh: perf.wm.o: alias /profiler/report to reporter.py [operations/puppet] - 10https://gerrit.wikimedia.org/r/122629 
[02:16:53] <grrrit-wm>	 (03PS2) 10Ori.livneh: perf.wm.o: alias /profiler/report to reporter.py [operations/puppet] - 10https://gerrit.wikimedia.org/r/122629 
[02:17:17] <grrrit-wm>	 (03PS1) 10Ori.livneh: Provide sensible configuration defaults [operations/software/mwprof/reporter] - 10https://gerrit.wikimedia.org/r/122630 
[02:17:31] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032 V: 032] Provide sensible configuration defaults [operations/software/mwprof/reporter] - 10https://gerrit.wikimedia.org/r/122630 (owner: 10Ori.livneh)
[02:18:14] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032 V: 032] perf.wm.o: alias /profiler/report to reporter.py [operations/puppet] - 10https://gerrit.wikimedia.org/r/122629 (owner: 10Ori.livneh)
[02:24:53] <grrrit-wm>	 (03PS1) 10Ori.livneh: decom report.py from noc.wikimedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/122632 
[02:26:41] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] decom report.py from noc.wikimedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/122632 (owner: 10Ori.livneh)
[02:28:27] <ori>	 springle: that's OK
[02:29:06] <springle>	 ori: thanks
[02:29:21] <ori>	 thank you
[02:32:39] <grrrit-wm>	 (03PS1) 10Ori.livneh: Remove reference to old cgi-bin files on noc [operations/puppet] - 10https://gerrit.wikimedia.org/r/122633 
[02:34:06] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Remove reference to old cgi-bin files on noc [operations/puppet] - 10https://gerrit.wikimedia.org/r/122633 (owner: 10Ori.livneh)
[02:36:58] <grrrit-wm>	 (03CR) 10Springle: [C: 04-1] "It would need a maintainer." [operations/puppet] - 10https://gerrit.wikimedia.org/r/111152 (owner: 10Diederik)
[02:41:24] <grrrit-wm>	 (03PS5) 10Springle: Update to latest jquery point releases [operations/software] - 10https://gerrit.wikimedia.org/r/118940 (owner: 10Reedy)
[02:41:51] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Update to latest jquery point releases [operations/software] - 10https://gerrit.wikimedia.org/r/118940 (owner: 10Reedy)
[02:42:10] <grrrit-wm>	 (03PS3) 10Springle: Define $errstr [operations/software] - 10https://gerrit.wikimedia.org/r/118949 (owner: 10Reedy)
[02:42:16] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Define $errstr [operations/software] - 10https://gerrit.wikimedia.org/r/118949 (owner: 10Reedy)
[02:43:11] <grrrit-wm>	 (03PS2) 10Springle: Replace & with &amp; in ganglia url [operations/software] - 10https://gerrit.wikimedia.org/r/120749 (owner: 10Reedy)
[02:43:19] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Replace & with &amp; in ganglia url [operations/software] - 10https://gerrit.wikimedia.org/r/120749 (owner: 10Reedy)
[02:51:55] <grrrit-wm>	 (03PS2) 10Springle: Stylize and function parameter and return type docblocks [operations/software] - 10https://gerrit.wikimedia.org/r/119685 (owner: 10Reedy)
[02:53:37] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Stylize and function parameter and return type docblocks [operations/software] - 10https://gerrit.wikimedia.org/r/119685 (owner: 10Reedy)
[02:56:14] <grrrit-wm>	 (03PS3) 10Springle: Use minified CSS files [operations/software] - 10https://gerrit.wikimedia.org/r/118951 (owner: 10Reedy)
[02:56:46] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Use minified CSS files [operations/software] - 10https://gerrit.wikimedia.org/r/118951 (owner: 10Reedy)
[03:05:26] <grrrit-wm>	 (03PS1) 10Springle: Minified file name typo. [operations/software] - 10https://gerrit.wikimedia.org/r/122634 
[03:05:46] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Minified file name typo. [operations/software] - 10https://gerrit.wikimedia.org/r/122634 (owner: 10Springle)
[03:10:06] <grrrit-wm>	 (03PS1) 10Springle: S7 db1034 upgrade to 5.5.36 running smoothly. Increase to full load. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122635 
[03:10:44] <grrrit-wm>	 (03CR) 10Springle: [C: 032] S7 db1034 upgrade to 5.5.36 running smoothly. Increase to full load. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122635 (owner: 10Springle)
[03:10:51] <grrrit-wm>	 (03Merged) 10jenkins-bot: S7 db1034 upgrade to 5.5.36 running smoothly. Increase to full load. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122635 (owner: 10Springle)
[03:11:33] <logmsgbot>	 !log springle synchronized wmf-config/db-eqiad.php  's7 db1034 full steam'
[03:11:39] <morebots>	 Logged the message, Master
[03:35:51] <Gloria>	 ori: I guess I'm a bit fascinated by the... bureaucracy forming, re: SWAT.
[03:36:04] <Gloria>	 It's playful bureaucracy or something, but still, eh.
[04:52:23] <grrrit-wm>	 (03PS2) 10Springle: MariaDB monitoring for disk and process(es) [operations/puppet] - 10https://gerrit.wikimedia.org/r/121676 
[04:54:04] <grrrit-wm>	 (03CR) 10Springle: [C: 032] MariaDB monitoring for disk and process(es) [operations/puppet] - 10https://gerrit.wikimedia.org/r/121676 (owner: 10Springle)
[06:10:16] <grrrit-wm>	 (03PS1) 10Springle: Class for wmf-mariadb build, versions 5.5 and 10.0. Roll out 5.5.36 on db1044. [operations/puppet] - 10https://gerrit.wikimedia.org/r/122643 
[06:16:19] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Class for wmf-mariadb build, versions 5.5 and 10.0. Roll out 5.5.36 on db1044. [operations/puppet] - 10https://gerrit.wikimedia.org/r/122643 (owner: 10Springle)
[06:25:16] <paravoid>	 ooh, mariadb 10
[06:25:17] <paravoid>	 me likes
[06:30:15] <grrrit-wm>	 (03PS1) 10Springle: Correct package name typo. [operations/puppet] - 10https://gerrit.wikimedia.org/r/122645 
[06:32:09] <grrrit-wm>	 (03CR) 10Springle: [C: 032] Correct package name typo. [operations/puppet] - 10https://gerrit.wikimedia.org/r/122645 (owner: 10Springle)
[06:54:35] <_joe_>	 hi all :) new team member here
[06:58:44] <paravoid>	 _joe_: hi!
[06:59:45] <_joe_>	 paravoid: hi there!
[07:08:25] <springle>	 _joe_: as you may gather from the stampede of greetings, this is a busy time of day ;) welcome
[07:08:42] <_joe_>	 springle: yeah I figured that out
[07:09:18] <_joe_>	 springle: not everybody's in europe and not everybody has to wake up at 7 to take their siblings to school
[07:11:49] <springle>	 heh
[07:13:17] <yurik>	 springle, thx!
[07:15:47] <yurik>	 springle, are you the person to bug with the https://gerrit.wikimedia.org/r/#/c/119990/ ?
[07:17:39] <springle>	 yurik: looking
[07:21:35] <springle>	 yurik: I think Reedy will normally do that deploy
[07:21:43] <paravoid>	 see my mail on ops
[07:21:52] <paravoid>	 and let's just stop this flood of irc pings & emails
[07:22:34] <springle>	 fine with me
[07:33:57] <grrrit-wm>	 (03CR) 10Springle: [C: 031] "Gerrit 122588 is done and sanitarium instance restarted." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119990 (owner: 10Reedy)
[07:34:23] <mutante>	 _joe_: welcome!!
[07:34:44] <mutante>	 going to start onboarding you:)
[07:34:56] <matanya>	 morning mutante tim reverted the CUcheck :/
[07:35:15] <matanya>	 welcome _joe_ 
[07:35:37] <mutante>	 matanya: do you know why?
[07:35:41] <matanya>	 yes
[07:35:48] <matanya>	 https://gerrit.wikimedia.org/r/#/c/74591/11/manifests/misc/maintenance.pp
[07:35:53] <matanya>	 comments from hoo
[07:35:59] <matanya>	 redoing now
[07:36:46] <mutante>	 ah, logging to /home, hah
[07:37:08] <mutante>	 well, good.. because at one point i said something like "all the scripts are logging to a different place"
[07:37:21] <mutante>	 and Reedy made a patch to make that consistent
[07:37:33] <mutante>	 had suggested /var/log/mediawiki 
[07:37:49] <matanya>	 ok, will point it to there
[07:37:53] <matanya>	 what about the user?
[07:38:29] <mutante>	 i think apache.. 
[07:38:36] <mutante>	 compare the other crons
[07:39:26] <grrrit-wm>	 (03CR) 10Springle: "Btw, this was dropped from sanitarium and labsdbs. It was never accessible to labs users." [operations/puppet] - 10https://gerrit.wikimedia.org/r/122610 (owner: 10Reedy)
[07:39:37] <matanya>	 mwdeploy
[07:39:55] <mutante>	 purge_securepoll uses apache
[07:40:10] <matanya>	 right
[07:40:15] <mutante>	 csteipp agreed afair
[07:40:21] <matanya>	 which one is better?
[07:41:55] <mutante>	 not sure, maybe none and they could all use the same
[07:42:25] <mutante>	 but i'd have to look closer and onboarding joe now
[07:43:05] <mutante>	 i know, you wanted to kill hume ..would have been nice:) thanks for pushing for that merge
[07:44:39] <hashar>	 hello
[07:45:39] <grrrit-wm>	 (03CR) 10Dzahn: "thanks Sean, adding Ottomata" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122406 (owner: 10Dzahn)
[07:48:51] <mutante>	 nuria: hi, do you know about research data used by analytics?
[07:49:06] <mutante>	 researchdb on db67 that is
[07:49:34] <nuria>	 no, I am afraid not, that must be from one of the analysts
[07:50:19] <mutante>	 alright, Sean mailed wmfresearch@
[07:50:29] <mutante>	 does that sound right?
[07:51:30] <nuria>	 and cc analytics
[07:51:37] <nuria>	 analytics internal
[07:51:47] <nuria>	 is there an issue (like a long running query?)
[07:53:16] <mutante>	 nuria: the issue is we would like to shutdown the database because it's in Tampa
[07:53:27] <mutante>	 but not a problem like query 
[07:53:50] <mutante>	 we want to know what should happen to the data etc
[07:54:09] <mutante>	 thanks!
[07:54:18] <mutante>	 springle: ^
[07:54:19] <springle>	 mutante: had one response via wmfresearch@ already. let them have a day to all chime in
[07:54:32] <nuria>	 I see, then an e-mail to analitics-internal would be best
[07:54:34] <mutante>	 cool.. yep
[07:54:37] <springle>	 that list seems to usually work
[07:54:39] <springle>	 oh ok
[07:55:26] <matanya>	 i will need someone from platform for this
[07:55:34] <nuria>	 better both lists as research & dev are intermingled on analytics team 
[07:56:34] <matanya>	 springle: any comment on: https://etherpad.wikimedia.org/p/what_is_left_in_tampa ?
[07:57:06] <mutante>	 matanya: that's the same thing.. re:db67, no it's not ok yet
[07:57:13] <mutante>	 because analytics data has been found
[07:57:33] <springle>	 nuria: done
[07:57:47] <nuria>	 thank youuu
[07:57:51] <mutante>	 well, ok, there are other ones. regarding "es" and amaranth .. true.. yes
[07:59:17] <springle>	 what a horrible set of colors to read through...
[08:00:24] <mutante>	 hehe, just search your own name i think:)
[08:00:40] <mutante>	 just the lines where it mentions you
[08:00:42] <springle>	 matanya: amaranth must stay on 12th. es is already answered
[08:01:29] <springle>	 i'd include that comment on amaranth.. but firefox is playing up on etherpad and I can't edit :)
[08:01:54] <springle>	 it must be scared of the colors
[08:09:09] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "What you said :-)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122599 (owner: 10BryanDavis)
[08:09:20] <grrrit-wm>	 (03Merged) 10jenkins-bot: Beta: set math storage directory to NFS share [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122599 (owner: 10BryanDavis)
[08:10:01] <matanya>	 springle: refresh and it might work. thanks anyway
[08:28:12] <grrrit-wm>	 (03PS1) 10Hashar: beta: adjust $wgCaptchaDirectory [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122657 
[08:29:30] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] beta: adjust $wgCaptchaDirectory [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122657 (owner: 10Hashar)
[08:29:39] <grrrit-wm>	 (03Merged) 10jenkins-bot: beta: adjust $wgCaptchaDirectory [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122657 (owner: 10Hashar)
[08:30:52] <grrrit-wm>	 (03PS1) 10Matanya: purge-checkuser: puppetize [operations/puppet] - 10https://gerrit.wikimedia.org/r/122658 
[08:43:11] <mutante>	 !log solr3 - delete salt key, puppet cert
[08:43:16] <morebots>	 Logged the message, Master
[08:44:03] <mutante>	 !log solr1/2 - revoke puppet certs
[08:44:06] <mutante>	 apergos: fixed
[08:44:07] <morebots>	 Logged the message, Master
[08:46:32] <apergos>	 yay
[08:49:30] <mutante>	 you are right, "stafford" lacks tracking
[08:49:35] <mutante>	 adding it
[08:58:48] <grrrit-wm>	 (03PS1) 10Hashar: beta: CentralAuthVerbose logs to centralauth.log [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122661 
[08:58:58] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] beta: CentralAuthVerbose logs to centralauth.log [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122661 (owner: 10Hashar)
[08:59:14] <grrrit-wm>	 (03Merged) 10jenkins-bot: beta: CentralAuthVerbose logs to centralauth.log [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122661 (owner: 10Hashar)
[09:00:34] <grrrit-wm>	 (03CR) 10Hoo man: [C: 04-1] "Redundant to https://gerrit.wikimedia.org/r/122617" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122658 (owner: 10Matanya)
[09:01:21] <grrrit-wm>	 (03Abandoned) 10Matanya: purge-checkuser: puppetize [operations/puppet] - 10https://gerrit.wikimedia.org/r/122658 (owner: 10Matanya)
[09:02:20] <matanya>	 mutante: can now merge : https://gerrit.wikimedia.org/r/#/c/122617/
[09:02:32] <matanya>	 thanks hoo
[09:09:20] <Bsadowski1>	 hoo is matanya
[09:09:28] <matanya>	 :)
[09:28:16] * matanya  is so tempted to push a patch giving access to Giuseppe but resists

[09:31:33] <grrrit-wm>	 (03PS1) 10Hashar: contint: get rid of misc::pbuilder on slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/122707 
[09:35:33] <grrrit-wm>	 (03CR) 10Hashar: [C: 031 V: 032] "Cherry picked on integration puppetmaster integration-puppetmaster.eqiad.wmflabs and applied on both slaves (integration-slave1001 and 100" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122707 (owner: 10Hashar)
[09:35:40] <grrrit-wm>	 (03CR) 10Matanya: contint: get rid of misc::pbuilder on slaves (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/122707 (owner: 10Hashar)
[09:37:56] <grrrit-wm>	 (03CR) 10Hashar: contint: get rid of misc::pbuilder on slaves (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/122707 (owner: 10Hashar)
[09:38:06] * hashar  looks at https://integration.wikimedia.org/ci/job/operations-debs-pybal-debian-glue/7/consoleFull

[09:42:04] <matanya>	 failed :/
[09:46:24] <springle>	 cd
[09:46:27] <springle>	 er
[09:46:34] <springle>	 hehe
[09:47:36] <matanya>	 wrong window i guess :P
[09:48:50] <grrrit-wm>	 (03PS1) 10Matanya: pbuilder: remove labs builder, redundant per https://gerrit.wikimedia.org/r/#/c/122707/1/modules/contint/manifests/packages/labs.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/122708 
[09:49:09] <matanya>	 hashar: ^
[09:49:58] <mutante>	 shutdown -h now
[09:49:59] <hashar>	 matanya: well one might still want to use the role class to have pbuilder / cowbuilder images
[09:50:04] <hashar>	 mutante: wrong term
[09:50:33] <matanya>	 hashar: not sure labs is the place for that
[09:50:47] <mutante>	 hashar: joking about wrong terms and decom:)
[09:50:51] <matanya>	 might be better to have a host specifily for that
[09:51:13] <mutante>	 i did that once on the wrong terminal though, hehe
[09:51:35] <matanya>	 spof on any case ?
[09:51:43] <mutante>	 since then i don't let a mouse change window focus without clicking
[09:57:56] <grrrit-wm>	 (03PS1) 10Hashar: contint: directory to hold debian-glue packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/122712 
[09:58:36] <hashar>	 matanya: the place for what?
[09:58:48] <matanya>	 for building images
[09:59:04] <matanya>	 packages
[09:59:09] <hashar>	 to build debian package I need to grant jenkins some sudo rights which is not really something we want on production
[09:59:18] <matanya>	 right
[09:59:30] <matanya>	 as said, a dedicated labs machine
[10:03:18] <grrrit-wm>	 (03PS2) 10Hashar: contint: directory to hold debian-glue packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/122712 
[10:13:59] <grrrit-wm>	 (03CR) 10Hashar: [C: 031 V: 032] "Fixed up permissions." [operations/puppet] - 10https://gerrit.wikimedia.org/r/122712 (owner: 10Hashar)
[10:21:01] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] beta: lower # of procs on jobrunner [operations/puppet] - 10https://gerrit.wikimedia.org/r/122436 (owner: 10Hashar)
[10:22:19] <matanya>	 akosiaris: any comment on https://etherpad.wikimedia.org/p/what_is_left_in_tampa ?
[10:22:35] <hashar>	 off for lunch
[10:31:37] <akosiaris>	 matanya: yeah, brewster is going to be finished this week
[10:45:55] <mark>	 \o/
[10:47:09] <matanya>	 mind commenting there akosiaris ? mark, you too please :)
[10:52:40] <mark>	 nothing to comment :)
[10:53:27] <matanya>	 mark: no even  "show this to Mark?" :)
[10:53:57] <mark>	 i can just remove that right now
[10:53:57] <matanya>	 or on this change: https://gerrit.wikimedia.org/r/#/c/108314/ ?
[10:54:30] <matanya>	 so atleast manutius can start moving to netmon1001 ?
[10:54:38] <matanya>	 (torrus)
[10:54:43] <mark>	 i might do that next week
[10:55:41] <matanya>	 i'll remind myself to nugde you about it. i'll get the nudge budge
[10:58:26] <mutante>	 mark: that was about the stuff above that.. "remove the static routes from pmtpa routers "
[10:58:46] <mutante>	 doing that would unblock shutting down capella
[10:59:10] <mutante>	 because Faidon said "We can decom if we need to reduce our footprint, but we'll need to remove the static routes from pmtpa routers or we'll otherwise blackhole traffic."
[10:59:37] <mutante>	 tries reducing footprint:)
[11:00:30] <mutante>	 i was thinking maybe chasemp is interested in it too
[11:01:19] <mark>	 already done
[11:01:32] <matanya>	 the routes ?
[11:01:40] <mutante>	 oh, ah:) cool
[11:01:48] <mutante>	 just saw update on etherpad
[11:02:04] <matanya>	 oh, yes :)
[11:02:05] <mutante>	 thx, looks like we can go on with capella then
[11:02:33] <matanya>	 need to verify nothing points there before
[11:02:43] <matanya>	 i'll push decom patch
[11:03:05] <mutante>	 yes, thanks, add me 
[11:04:36] <mark>	 and making another machine a ganglia aggregator is trivial too, right
[11:04:46] <mark>	 just pick any misc server that likely will stay on the 12th floor longest
[11:05:21] <mark>	 fenari or whatever
[11:07:22] <mutante>	 oh, even fenari could be? i didnt really know how trivial it was. ok
[11:07:44] <mutante>	 btw, labs ganglia broke was an issue today
[11:08:26] <grrrit-wm>	 (03PS1) 10Matanya: capella: decom [operations/puppet] - 10https://gerrit.wikimedia.org/r/122718 
[11:09:10] <mutante>	 well, ok, i see RobH commented as well "if all the misc servers are shut down, then the misc ganglia aggregator can shut down"
[11:09:49] <mutante>	 where fenari is a misc, so, gotcha
[11:11:49] <grrrit-wm>	 (03CR) 10Mark Bergsma: "Nice, just some very minor comments. Once you correct those, I'll +1 or +2." (035 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/115345 (owner: 10Jkrauska)
[11:12:36] <grrrit-wm>	 (03PS1) 10Matanya: capella: decom [operations/dns] - 10https://gerrit.wikimedia.org/r/122719 
[11:17:33] <nuria>	 anyone that can help with errors running puppet on wikimetrics staging? (wikimetrics-staging1.eqiad.wmflabs) we just recently migrated it and maybe there is something we missed
[11:17:50] <matanya>	 what is the error nuria ?
[11:18:08] <nuria>	 err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class passwords::wikimetrics for wikimetrics-staging1.eqiad.wmflabs at /etc/puppet/manifests/role/wikimetrics.pp:40 on node wikimetrics-staging1.eqiad.wmflabs
[11:18:08] <nuria>	 warning: Not using cache on failed catalog
[11:18:08] <nuria>	 err: Could not retrieve catalog; skipping run
[11:18:08] <nuria>	 root@wikimetrics-staging1:/var/lib/git/operations#
[11:18:19] <nuria>	 when running sudo puppetd -tv
[11:18:38] <matanya>	 did you include that class ?
[11:18:39] <mutante>	 nuria: that sounds like it is missing a passwords class in the labs/private repo
[11:18:54] <mutante>	 in prod. we have actual passwords in the private repo
[11:19:10] <mutante>	 in labs there is labs/private for making it possible to test
[11:19:17] <mutante>	 but with fake passwords more or less
[11:20:10] <mutante>	 so you have some "include passwords::.." somewhere, and that class is missing
[11:20:20] <nuria>	 this is actually "self hosted puppetmaster"
[11:20:48] <mutante>	 was it self hosted before the migration as well?
[11:20:52] <nuria>	 yes
[11:21:13] <nuria>	 so i imagine is setup a little differently from labs?
[11:22:11] <mutante>	 it sounds like you just need one more file with that password class
[11:22:23] <mutante>	 maybe it was added manually on the old instance
[11:22:36] <mutante>	 and not copied during migration
[11:23:24] <nuria>	 ok, i shall ask otto then 
[11:23:25] <mutante>	 is the old instance still alive? 
[11:23:31] <nuria>	 mmm.. let me see
[11:23:44] <mutante>	 you could ask andrewbogott_afk nicely if he can give you data
[11:23:54] <mutante>	 from old instance, the entire /etc/puppet/
[11:24:05] <mutante>	 in case it's down but "mothballed"
[11:24:50] <mutante>	 or create a new class passwords::wikimetrics in the right place
[11:25:31] <nuria>	 but i do not know the right passwords so i would not help me much...i will try to ask nicely
[11:25:39] <nuria>	 i think old instance might be alive wikimetrics.pmtpa.wmflabs
[11:25:53] <mutante>	 let's check the labs/private repo
[11:26:11] <mutante>	 it's not clear to me yet if this was in a repo or manual setup on old instance
[11:26:46] <mutante>	 ah, here we go
[11:26:48] <mutante>	 manifests/role/wikimetrics.pp:# passwords::wikimetrics is not a real class checked in to any repository.
[11:27:05] <nuria>	 so what is it?
[11:27:22] <mutante>	 something people just put into /etc/puppet somewhere
[11:27:31] <mutante>	 on the instance with self::hosted puppetmaster
[11:27:40] <nuria>	 ah just a file with pw
[11:27:48] <mutante>	 with a class that sets pw's
[11:27:57] <nuria>	 right.ok
[11:28:04] <mutante>	 lemme show you other examples that _are_ checked in
[11:28:41] <nuria>	 i see so you manage pw normaly with a private git depot only available on labs
[11:28:46] <mutante>	 https://gerrit.wikimedia.org/r/#/c/119871/1/modules/passwords/manifests/init.pp
[11:29:15] <mutante>	 see this https://gerrit.wikimedia.org/r/#/q/project:labs/private,n,z
[11:29:27] <mutante>	 this is how people do that on labs to solve those issues but have it in repo
[11:29:40] <mutante>	 but the password isnt private
[11:29:47] <mutante>	 so it's for testing the puppet code
[11:30:09] <nuria>	 i think we must have a similar one in vagrant as our puppet code works there
[11:30:13] <mutante>	 if you need to keep it private, then you need to keep doing it as it was before and restrict the labs project
[11:31:23] <nuria>	 but i get it thank you. likely ours should be private , i will ask otto as the best set up
[11:31:27] <mutante>	 as before = self hosted puppetmaster and manually putting it there
[11:31:38] <mutante>	 yw, yes
[11:31:41] <nuria>	 right as in "creating file on machine"
[11:31:45] <mutante>	 yes
[11:31:57] <nuria>	 very enlightening thank you
[11:32:26] <mutante>	 and making sure others don't get to it who should not,which would make it a restricted project
[11:33:04] <mutante>	 all depends what is behind that password
[11:33:30] <grrrit-wm>	 (03CR) 10Steinsplitter: "NONONO... now somone gas deleted all uloads via DB..... http://commons.wikimedia.beta.wmflabs.org/w/index.php?title=Special:ListFiles/Stei" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122084 (owner: 10Mattflaschen)
[11:34:50] <grrrit-wm>	 (03CR) 10Steinsplitter: [C: 04-1] Have Commons on Beta Labs use $stdlogo [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122084 (owner: 10Mattflaschen)
[11:45:29] <mutante>	 nuria: or you can try just commenting line 40 in /etc/puppet/manifests/role/wikimetrics.pp so it simple skips the file and see what happens..depending on your module and what're you're testing you can still see the rest of the puppet run then.. even though the app will have no db access due to missing pass.. well. that was just a P.S., bbiaw
[11:46:23] <nuria>	 ya.. it would not work that great then so i guess i just need to wait a bit for people to wake up in the east coast
[11:46:33] <mutante>	 (since you're on self::hosted puppetmaster it doesn't need a patch, just change it and puppetd -tv)
[11:46:49] <mutante>	 ok, /me nods
[11:59:24] <nuria>	 thanks mutante for your help, i hope to have this fix today
[12:01:59] <grrrit-wm>	 (03PS1) 10Dzahn: add shell account for Giuseppe (oblivian) [operations/puppet] - 10https://gerrit.wikimedia.org/r/122728 
[12:24:36] <grrrit-wm>	 (03PS2) 10Dzahn: add shell account for Giuseppe (oblivian) [operations/puppet] - 10https://gerrit.wikimedia.org/r/122728 
[12:26:01] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] "Giuseppe, test, do you get mail already when i do this?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122728 (owner: 10Dzahn)
[12:29:48] <grrrit-wm>	 (03CR) 10Dzahn: [C: 04-1] "ok, cool, so: i delibarately put a fixme in there, to: show inline comments, give you something to amend to and try git-review and confirm" (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/122728 (owner: 10Dzahn)
[12:44:06] <grrrit-wm>	 (03PS1) 10Giuseppe Lavagetto: Remove the fixme, add oblivian to admins. [operations/puppet] - 10https://gerrit.wikimedia.org/r/122774 
[12:47:31] <grrrit-wm>	 (03CR) 10Matanya: [C: 04-1] "duplicate of https://gerrit.wikimedia.org/r/#/c/122728/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122774 (owner: 10Giuseppe Lavagetto)
[12:49:40] <mutante>	 matanya: should creating account and adding it to roots be 1 or 2 patches?:) 
[12:49:43] <_joe_>	 matanya: yep that was my first attempt at using gerrit, sorry
[12:49:48] <mutante>	 be back in a little
[12:50:09] <mutante>	 (we can do it either way)
[12:50:13] <matanya>	 oh, sorry. _joe_ didn't note it was your patch
[12:50:13] <mutante>	 no worries
[12:50:47] <matanya>	 mutante: i'll ignore that comment for well known reasons :)
[12:50:48] <mutante>	 matanya: trying to combine the actual account creation and showing how to use gerrit into the same thing again:)
[12:50:50] <mutante>	 brb
[12:51:07] <mutante>	 matanya: no, i really asked
[12:51:21] <matanya>	 in my opinion, 2. you know that
[12:51:51] <grrrit-wm>	 (03CR) 10Hashar: "I think we should keep that role class, that let one easily setup a package building environment on a labs instance." [operations/puppet] - 10https://gerrit.wikimedia.org/r/122708 (owner: 10Matanya)
[12:52:01] <mutante>	 then it's not duplicate and joe should not abandon anything
[12:52:21] <grrrit-wm>	 (03Abandoned) 10Matanya: pbuilder: remove labs builder, redundant per https://gerrit.wikimedia.org/r/#/c/122707/1/modules/contint/manifests/packages/labs.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/122708 (owner: 10Matanya)
[12:52:21] <mutante>	 just merge them in order and that they have dependencies is good in this case
[12:52:49] <_joe_>	 mutante: ok good
[12:53:05] * matanya  stops bothering others

[12:54:17] <_joe_>	 matanya: well, it's just I did not read the manual for gerrit and/or git-review more than a bunch of tutorials on the wiki so I didn't fully understand what was going on
[12:55:04] <matanya>	 it is totally fine _joe_ really. i was probably the worst contributer when i started volunteering
[12:55:22] <matanya>	 i hope it changed since then
[12:55:45] <grrrit-wm>	 (03CR) 10QChris: [C: 04-1] "Looks good to me." [operations/puppet] - 10https://gerrit.wikimedia.org/r/121546 (owner: 10Ottomata)
[12:56:56] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 031] "This actually adds also my user to the admins::roots class, so it should be a separate change (although that was unintentional)." [operations/puppet] - 10https://gerrit.wikimedia.org/r/122774 (owner: 10Giuseppe Lavagetto)
[12:57:31] <grrrit-wm>	 (03CR) 10Matanya: [C: 031] Remove the fixme, add oblivian to admins. [operations/puppet] - 10https://gerrit.wikimedia.org/r/122774 (owner: 10Giuseppe Lavagetto)
[13:00:39] <_joe_>	 matanya: /win 19
[13:00:43] <_joe_>	 oh sorry
[13:01:05] <grrrit-wm>	 (03PS3) 10Dzahn: add shell account for Giuseppe (oblivian) [operations/puppet] - 10https://gerrit.wikimedia.org/r/122728 
[13:01:05] <mutante>	 irssi user:)
[13:01:27] <_joe_>	 yep
[13:01:38] <_joe_>	 and via screen on a remote machine
[13:01:42] <mutante>	 same here
[13:01:48] <_joe_>	 so lag sometimes fools me
[13:02:00] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] add shell account for Giuseppe (oblivian) [operations/puppet] - 10https://gerrit.wikimedia.org/r/122728 (owner: 10Dzahn)
[13:05:29] <mutante>	 my bad
[13:05:37] <mutante>	 reusing topic branches
[13:21:22] <grrrit-wm>	 (03PS4) 10Dzahn: add shell account for Giuseppe (oblivian) [operations/puppet] - 10https://gerrit.wikimedia.org/r/122728 
[13:24:55] <Nemo_bis>	 question for ops by mozilla at https://bugzilla.mozilla.org/show_bug.cgi?id=758857
[13:25:17] <Nemo_bis>	 uuh italians
[13:25:28] <grrrit-wm>	 (03CR) 10Matanya: [C: 031] add shell account for Giuseppe (oblivian) [operations/puppet] - 10https://gerrit.wikimedia.org/r/122728 (owner: 10Dzahn)
[13:26:41] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] "key confirmed with gpg and on office wiki" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122728 (owner: 10Dzahn)
[13:28:55] <grrrit-wm>	 (03CR) 10Manybubbles: [C: 031] "Lets sync it on Wednesday when we switch most of group1 to primary." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122571 (owner: 10Chad)
[13:29:07] <grrrit-wm>	 (03PS2) 10Giuseppe Lavagetto: Remove the fixme, add oblivian to admins. [operations/puppet] - 10https://gerrit.wikimedia.org/r/122774 
[13:29:14] <matanya>	 hmm, do we have any memcaches on lucid ?
[13:30:20] <grrrit-wm>	 (03PS1) 10Hashar: role::beta::uploadservice to allow port 80 [operations/puppet] - 10https://gerrit.wikimedia.org/r/122786 
[13:31:18] <grrrit-wm>	 (03PS3) 10Dzahn: add Giuseppe (oblivian) to admins::roots [operations/puppet] - 10https://gerrit.wikimedia.org/r/122774 (owner: 10Giuseppe Lavagetto)
[13:33:46] <grrrit-wm>	 (03CR) 10Hashar: [C: 031 V: 032] "Cherry picked on integration-puppetmaster.eqiad.wmflabs. Role applied on deployment-upload.eqiad.wmflabs which restored thumb/upload servi" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122786 (owner: 10Hashar)
[13:34:09] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 031] "The commit message is much clearer now, thanks." [operations/puppet] - 10https://gerrit.wikimedia.org/r/122774 (owner: 10Giuseppe Lavagetto)
[13:36:06] <grrrit-wm>	 (03CR) 10Matanya: [C: 031] add Giuseppe (oblivian) to admins::roots [operations/puppet] - 10https://gerrit.wikimedia.org/r/122774 (owner: 10Giuseppe Lavagetto)
[13:37:07] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] "letting you merge it for demonstration purposes" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122774 (owner: 10Giuseppe Lavagetto)
[13:38:57] <grrrit-wm>	 (03CR) 10Hashar: [C: 031] "Hi!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122774 (owner: 10Giuseppe Lavagetto)
[13:39:32] <grrrit-wm>	 (03CR) 10Dzahn: "i'll merge it for you on the puppetmaster afterwards. this is a sanity check and chicken/egg problem. you dont have access yet to give you" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122774 (owner: 10Giuseppe Lavagetto)
[13:43:04] <mutante>	 !log adding oblivion to ops and wmf LDAP groups
[13:43:10] <morebots>	 Logged the message, Master
[14:00:53] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 04-1] "This is a noop. However it is a noop for convoluted reasons. So what happens here is that the @ in-front of those variables squashes the w" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119253 (owner: 10Matanya)
[14:04:07] <hashar>	 !log Gerrit flushed a few caches related to user accounts / LDAP
[14:04:12] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] "Indeed a noop. Merging" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122131 (owner: 10Hashar)
[14:04:13] <morebots>	 Logged the message, Master
[14:05:18] <hashar>	 akosiaris: thank you :-]
[14:05:31] <mutante>	 yes, +1:)
[14:05:51] <mutante>	 thanks for that patch
[14:06:46] <ottomata>	 akosiaris: could you take a quick look at this when you get a chance?
[14:06:47] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/121856/
[14:06:48] <ottomata>	 shoudl be easy
[14:06:54] <grrrit-wm>	 (03PS5) 10Matanya: gerrit: qualify var [operations/puppet] - 10https://gerrit.wikimedia.org/r/119253 
[14:06:56] <ottomata>	 never added a bacula thing before, just want to make sure I did it right
[14:08:26] <matanya>	 ottomata: https://etherpad.wikimedia.org/p/Puppet3 can you please look at looging and anayitics related stuff there please?
[14:09:30] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] "I'll merge, but follow-up for a different commit to remove the extra network." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119677 (owner: 10Reedy)
[14:09:42] <grrrit-wm>	 (03PS11) 10Faidon Liambotis: Manage scap proxy rsync config in puppet [operations/puppet] - 10https://gerrit.wikimedia.org/r/119677 (owner: 10Reedy)
[14:09:48] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032 V: 032] Manage scap proxy rsync config in puppet [operations/puppet] - 10https://gerrit.wikimedia.org/r/119677 (owner: 10Reedy)
[14:10:35] <grrrit-wm>	 (03PS6) 10Faidon Liambotis: Add mw1161 and mw1201 as scap proxies for EQIAD row C and D [operations/puppet] - 10https://gerrit.wikimedia.org/r/119686 (owner: 10Reedy)
[14:10:42] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032 V: 032] Add mw1161 and mw1201 as scap proxies for EQIAD row C and D [operations/puppet] - 10https://gerrit.wikimedia.org/r/119686 (owner: 10Reedy)
[14:11:19] <grrrit-wm>	 (03Abandoned) 10Faidon Liambotis: Update docroot_dir_allows to use network::constants::mediawiki_appservers [operations/puppet] - 10https://gerrit.wikimedia.org/r/119668 (owner: 10Reedy)
[14:11:47] <grrrit-wm>	 (03PS1) 10Hashar: iptables.pp: retab to four spaces [operations/puppet] - 10https://gerrit.wikimedia.org/r/122788 
[14:13:45] <grrrit-wm>	 (03CR) 10Hashar: "That is merely to get rid of the tabs indentation. Since that file will eventually be dropped in favor of ferm, I don't think it is worth " [operations/puppet] - 10https://gerrit.wikimedia.org/r/122788 (owner: 10Hashar)
[14:14:43] <paravoid>	 hashar: they why is it worth retabing it?
[14:15:14] <hashar>	 paravoid: less false positives when gripping our manifests for tabulations :]
[14:15:15] <grrrit-wm>	 (03CR) 10Dzahn: "can you create a ticket for this with some more details ?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122621 (owner: 10Reedy)
[14:15:20] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 032] "Merging." [operations/puppet] - 10https://gerrit.wikimedia.org/r/122774 (owner: 10Giuseppe Lavagetto)
[14:16:06] <grrrit-wm>	 (03CR) 10Dzahn: "merging on palladium" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122774 (owner: 10Giuseppe Lavagetto)
[14:16:45] <mutante>	 _joe_: so.. running puppet now to watch it
[14:17:12] <ottomata>	 matanya: are strike-through items in that list already fixed?
[14:17:13] <_joe_>	 mutante: that should be automatically merged?
[14:17:16] <ottomata>	 i only see strike-through analytics things
[14:17:32] <matanya>	 yes, striked ones i fixed
[14:17:36] <mutante>	 _joe_: i did it for you by typing puppet-merge on the master
[14:17:39] <mutante>	 (palladium)
[14:17:48] <_joe_>	 ok
[14:17:53] <mutante>	 !log welcome new shell user oblivion 
[14:17:55] <mutante>	 notice: /Stage[main]/Accounts::Oblivian/Unixaccount[Giuseppe Lavagetto]/User[oblivian]/ensure: created
[14:17:56] <matanya>	 ottomata: line 38-40
[14:17:59] <morebots>	 Logged the message, Master
[14:18:00] <mutante>	 there you go
[14:18:23] <grrrit-wm>	 (03PS1) 10Hashar: misc/dsh.pp: retab and almost pass puppet lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/122789 
[14:18:33] <matanya>	 ottomata: and 68 i suspect
[14:18:40] <mutante>	 _joe_: try logging in on iron.wikimedia.org
[14:18:48] <matanya>	 79 too ?
[14:20:32] <grrrit-wm>	 (03CR) 10Matanya: [C: 031] "hate partial lints, but well :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122789 (owner: 10Hashar)
[14:21:00] <grrrit-wm>	 (03CR) 10Matanya: [C: 031] iptables.pp: retab to four spaces [operations/puppet] - 10https://gerrit.wikimedia.org/r/122788 (owner: 10Hashar)
[14:21:21] <paravoid>	 I don't like retabing either tbh
[14:21:44] <paravoid>	 you mess with the git history (blame) without fixing anything, it's kind of annoying
[14:21:54] <matanya>	 +1 on that
[14:22:00] <paravoid>	 but I don't care that much, if it helps you in some way
[14:22:14] <matanya>	 it helps to pass puppet-lint
[14:22:37] <matanya>	 but if you do a lint, do it to the end. my cent on this
[14:23:17] <mutante>	 but if you dont do them everybody comments on the mixed tabs/spaces
[14:23:22] <mutante>	 cant win
[14:23:55] <matanya>	 sure you can't. some changes are too big, some are too small ...
[14:24:59] <matanya>	 but honestly, i really don't like half way lints. i know, easier to merge that way, but please 
[14:26:38] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] "LGTM" [operations/puppet] - 10https://gerrit.wikimedia.org/r/121856 (owner: 10Ottomata)
[14:28:17] <matanya>	 paravoid: if you have anything to comment on https://etherpad.wikimedia.org/p/what_is_left_in_tampa i'd like to see it
[14:28:53] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: "I successfully logged into production. It works!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122774 (owner: 10Giuseppe Lavagetto)
[14:29:20] <grrrit-wm>	 (03PS1) 10Hashar: ganglia: tweak settings for labs in eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/122790 
[14:30:52] <hashar>	 paravoid: matanya for git blame one can use:  git blame -w .  That ignores spaces changes
[14:31:17] <matanya>	 yes, know. still :)
[14:31:23] <hashar>	 also git gui blame file.ext
[14:31:37] <hashar>	 then you can browse the previous commits by simply clicking , saving a bunch of time
[14:37:27] <matanya>	 i'm so going to do this one day : http://sjoeboo.github.io/blog/2012/05/31/getting-racktables-location-info-into-puppet/
[14:38:28] <_joe_>	 matanya: uhm, not *so* cool IMHO
[14:39:18] <matanya>	 you have access, i don't. so anything that brings racktables to puppet works for me :)
[14:39:26] <paravoid>	 one of these days I'd like to kill racktables :)
[14:39:32] <matanya>	 that too
[14:39:59] <_joe_>	 matanya: no don't get me wrong, I wasn't talking about the functionality, but the implementation
[14:40:18] <matanya>	 oh, that. yeah, somewhat ackward
[14:41:23] <_joe_>	 the idea that data are pulled from racktables and split into yaml files to be read by facter does seem clunky to me
[14:42:26] <matanya>	 racktables doesn't have any api. need to look for a better tool
[14:42:46] <_joe_>	 (also, I'd expect to have that kind of info ideally in the same tool I use as an ENC for puppet)
[14:43:02] <_joe_>	 matanya: never used racktables, honestly :)
[14:43:42] <matanya>	 it is one of the future plans for the foreman iirc
[14:43:55] <_joe_>	 matanya: my point is, there are a few projects out there that could be used as inventory tools and external node classifiers for puppet, if somebody gave tham a little love
[14:44:36] <matanya>	 yes, i agree on that. time is the main issue. no one can afford loving :)
[14:44:44] <_joe_>	 :)
[14:45:59] <mutante>	 !log added oblivion to root-auth-keys
[14:46:04] <morebots>	 Logged the message, Master
[14:46:13] <mutante>	 _joe_: so..that was a commit in the private repo
[14:46:24] <mutante>	 you should check if you got mail about that too 
[14:46:31] <_joe_>	 ok
[14:46:39] <mutante>	 and then you can try logging in on "palladium"
[14:46:42] <mutante>	 going via iron
[14:47:05] <mutante>	 please use ProxyCommand
[14:47:06] <akosiaris>	 _joe_: welcome to the team :-)
[14:47:29] <_joe_>	 akosiaris: hi :) thanks!
[14:49:57] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] ganglia: tweak settings for labs in eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/122790 (owner: 10Hashar)
[14:59:07] <grrrit-wm>	 (03PS1) 10Hashar: Noop change (DO NOT SUBMIT) [operations/debs/ganglia] - 10https://gerrit.wikimedia.org/r/122833 
[15:01:01] <hashar>	 !log Gerrit super slow again :-(
[15:01:06] <morebots>	 Logged the message, Master
[15:01:45] <grrrit-wm>	 (03PS2) 10Hashar: Noop change (DO NOT SUBMIT) [operations/debs/ganglia] - 10https://gerrit.wikimedia.org/r/122833 
[15:03:08] <grrrit-wm>	 (03PS1) 10Matanya: ms10: decom [operations/puppet] - 10https://gerrit.wikimedia.org/r/122835 
[15:06:13] <logmsgbot>	 !log reedy updated /a/common to {{Gerrit|If3ca3d486}}: beta: adjust $wgCaptchaDirectory
[15:06:19] <morebots>	 Logged the message, Master
[15:06:20] <grrrit-wm>	 (03PS1) 10Reedy: Non wikipedias to 1.23wmf20 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122836 
[15:08:55] <grrrit-wm>	 (03PS1) 10Matanya: ms10: decom [operations/dns] - 10https://gerrit.wikimedia.org/r/122837 
[15:09:37] <manybubbles>	 ottomata: hey, so, what if I want to do one of those git-deploy doodads in labs to deploy some elasticsearch plugins, can I doodat?
[15:12:50] <ottomata>	 yup, the rsync module is public
[15:12:59] <ottomata>	 should work in the same way
[15:13:20] <ottomata>	 you could even use labs archiva if you want
[15:13:25] <ottomata>	 although i haven't updated it in a bit
[15:13:29] <ottomata>	 archiva.wmflabs.org
[15:13:51] <ottomata>	 btw, manybubbles, not sure if I sent this to you yet
[15:13:51] <ottomata>	 https://wikitech.wikimedia.org/wiki/Archiva
[15:14:10] <ottomata>	 its still a WIP, qchris and I are figuringout the best way to add archiva as your only rmote build repo
[15:14:11] <manybubbles>	 thanks!
[15:14:21] <ottomata>	 not sure if it is better to do in settings.xml or in project's pom.xml yet 
[15:15:04] <manybubbles>	 settings.xml
[15:15:19] <manybubbles>	 project pom's generally shouldn't have deployment information, I've been reading
[15:15:23] <manybubbles>	 can't find link
[15:26:30] <grrrit-wm>	 (03PS1) 10Ottomata: Using template_variables parameter to fix puppet lint problem [operations/puppet] - 10https://gerrit.wikimedia.org/r/122840 
[15:29:35] <grrrit-wm>	 (03PS2) 10Ottomata: Using template_variables parameter to fix puppet lint problem [operations/puppet] - 10https://gerrit.wikimedia.org/r/122840 
[15:29:42] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Using template_variables parameter to fix puppet lint problem [operations/puppet] - 10https://gerrit.wikimedia.org/r/122840 (owner: 10Ottomata)
[15:32:11] <ottomata>	 matanya: check your linter for those lines, i think should be fixed
[15:33:27] <matanya>	 ottomata: it is about puppet3, not lint :) will check
[15:33:35] <matanya>	 thanks for the patches
[15:35:00] <matanya>	 ottomata: it doesn't seem to me like ti fixed it
[15:35:09] <matanya>	 you are still calling that out of scope
[15:37:52] <ottomata>	 @template_variables isn't out of scope?
[15:37:52] <ottomata>	 is it?
[15:38:03] <ottomata>	 its passed to the define and the define renders the erb
[15:38:35] <matanya>	 where do you call filters.erbium.erb ?
[15:39:15] <ottomata>	 in define misc::udp2log::ins
[15:39:19] <ottomata>	 define misc::udp2log::instance
[15:41:35] <matanya>	 right, and the var is called in the scope of the class role::logging::udp2log::erbium 
[15:42:51] <ottomata>	 hm, no it is passed to define
[15:43:05] <ottomata>	         template_variables => {
[15:43:05] <ottomata>	             'fundraising_log_directory' => $fundraising_log_directory
[15:43:05] <ottomata>	         },
[15:43:23] <ottomata>	  <%= @template_variables['fundraising_log_directory'] %>/logs/bannerImpressions-sampled100.tsv.log
[15:46:16] <ottomata>	 heya	 cmjohnson1, what's the status of stat1003?
[15:46:48] <matanya>	 ottomata: i might not understand all that stuff
[15:47:01] <matanya>	 vut if you say it should be fixed, well :)
[15:47:04] <matanya>	 but
[15:47:14] <ottomata>	 well, i mean, @template_variables is in scope
[15:47:21] <ottomata>	 and I am only referencing that variable in the .erb
[15:47:23] <ottomata>	 soooo ja!
[15:47:25] <ottomata>	 should be ok
[15:47:39] <grrrit-wm>	 (03PS1) 10Cmjohnson: adding new dns entries for new misc servers/fixing typo and space issue [operations/dns] - 10https://gerrit.wikimedia.org/r/122845 
[15:48:10] <cmjohnson1>	 ottomata: noticed an eror on mgmt dns today. fixing that now...other than that ready for install
[15:50:17] <grrrit-wm>	 (03CR) 10Cmjohnson: [C: 032] adding new dns entries for new misc servers/fixing typo and space issue [operations/dns] - 10https://gerrit.wikimedia.org/r/122845 (owner: 10Cmjohnson)
[15:50:59] <ottomata>	 cmjohnson1: did we figure out partitioning / raid stuff?
[15:51:14] <cmjohnson1>	 raid5-lvs-gpt
[15:54:20] <qchris>	 manybubbles|away: Full ACK on having the deployment information in settings.xml. But the part I was discussing with ottomata was more about where to pull jars from.
[15:54:43] <qchris>	 manybubbles|away: so the <repositories> part. That should go into the pom.xml from my point of view.
[15:57:55] <ottomata>	 ok perfect, thanks qchris
[15:58:25] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] ms10: decom [operations/puppet] - 10https://gerrit.wikimedia.org/r/122835 (owner: 10Matanya)
[15:58:45] <grrrit-wm>	 (03CR) 10Dzahn: "when it's actually down" [operations/dns] - 10https://gerrit.wikimedia.org/r/122837 (owner: 10Matanya)
[16:03:51] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] "we spend more time talking about lint changes on IRC than on gerrit. once we finally got rid of those tabs it won't be that bad anymore.an" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122788 (owner: 10Hashar)
[16:05:09] <andrewbogott>	 !log switching wikitech to read-only, migrating to eqiad
[16:05:15] <morebots>	 Logged the message, Master
[16:05:48] <grrrit-wm>	 (03PS2) 10Dzahn: capella: decom [operations/puppet] - 10https://gerrit.wikimedia.org/r/122718 (owner: 10Matanya)
[16:08:30] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] capella: decom [operations/puppet] - 10https://gerrit.wikimedia.org/r/122718 (owner: 10Matanya)
[16:11:37] <icinga-wm>	 PROBLEM - puppetmaster https on virt0 is CRITICAL: Connection refused  
[16:11:58] <akosiaris>	 andrewbogott: is that ^ you ?
[16:12:04] <andrewbogott>	 akosiaris: yep!
[16:12:17] <icinga-wm>	 PROBLEM - HTTP on virt0 is CRITICAL: Connection refused  
[16:12:34] <andrewbogott>	 akosiaris: I'll hush up icinga
[16:13:41] <mutante>	 !log capella - disable puppet,revoke puppet cert,salt key..
[16:16:06] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Moved wikitech to virt1000 in eqiad. [operations/dns] - 10https://gerrit.wikimedia.org/r/122846 
[16:16:18] <andrewbogott>	 mark, paravoid, ^ ?
[16:16:29] <andrewbogott>	 It's a no-brainer, but not something I generally mess with
[16:17:19] <grrrit-wm>	 (03CR) 10Dzahn: "also check reverse record" [operations/dns] - 10https://gerrit.wikimedia.org/r/122846 (owner: 10Andrew Bogott)
[16:18:30] <andrewbogott>	 mutante: I think the reverse entry only points to virt0, which is correct.
[16:18:36] <andrewbogott>	 But, please double-check for me?
[16:19:17] <icinga-wm>	 PROBLEM - puppetmaster https on virt1000 is CRITICAL: Connection refused  
[16:19:37] <icinga-wm>	 PROBLEM - HTTP on virt1000 is CRITICAL: Connection refused  
[16:19:47] <andrewbogott>	 ^ that is also me :)
[16:20:21] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] "nevermind, lgtm" [operations/dns] - 10https://gerrit.wikimedia.org/r/122846 (owner: 10Andrew Bogott)
[16:21:21] <mutante>	 andrewbogott: looks ok, provived you already have the certificate on virt1000 etc
[16:22:15] <mutante>	 well, i should say "like before", not argueing about the "should it be A record or CNAME" part
[16:23:08] <andrewbogott>	 mutante, where would the cert be stored?  I think it's part of the big copy I'm making, but want to make sure...
[16:23:37] <mutante>	 andrewbogott: /etc/ssl/certs and /etc/ssl/private for cert and key.. 
[16:23:44] <mutante>	 should be done by install_certificate 
[16:23:58] * andrewbogott  checks

[16:24:16] <mutante>	 looks like you do
[16:24:23] <andrewbogott>	 yep, it's there already
[16:24:27] <mutante>	 wikitech.wikimedia.org.chained.pem
[16:24:34] <mutante>	 the .chained. version is made by puppet
[16:24:36] <mutante>	 with a simple cat
[16:24:41] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Moved wikitech to virt1000 in eqiad. [operations/dns] - 10https://gerrit.wikimedia.org/r/122846 (owner: 10Andrew Bogott)
[16:28:35] <mutante>	 !log capella - removed from monitoring, stored configs, stopping miredo daemon (IPv6)
[16:32:12] <greg-g>	 bah, blocked on work :)
[16:32:42] <mutante>	 because of which one:)
[16:33:16] <ottomata>	 wikitech down?
[16:33:23] <greg-g>	 wikitech down
[16:33:43] <greg-g>	 it's known/planned/all that, just wanted to do a quick edit before sending this email... ;)
[16:34:24] <mutante>	 i see:), yep
[16:35:35] <icinga-wm>	 RECOVERY - puppetmaster https on virt0 is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.160 second response time  
[16:37:42] <grrrit-wm>	 (03PS1) 10Dr0ptp4kt: Add OM and HTTPS support for 635-10. [operations/puppet] - 10https://gerrit.wikimedia.org/r/122851 
[16:38:45] <grrrit-wm>	 (03CR) 10Ottomata: [C: 031] Puppet masters in labs needs puppet::self::geoip [operations/puppet] - 10https://gerrit.wikimedia.org/r/121677 (owner: 10Hashar)
[16:38:55] <dr0ptp4kt>	 ^^ bblack, when you have a minute, simple vcl change for review
[16:43:22] <mutante>	 !log capella - shut down 
[16:44:55] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032] "Looks good, should we merge?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/112423 (owner: 10Matanya)
[16:45:35] <icinga-wm>	 PROBLEM - puppetmaster https on virt0 is CRITICAL: Connection refused  
[16:46:35] <icinga-wm>	 ACKNOWLEDGEMENT - puppetmaster https on virt0 is CRITICAL: Connection refused daniel_zahn scheduled migration
[16:49:14] <grrrit-wm>	 (03CR) 10Dzahn: "i was about to split this into 2 changes, leaving db67 alone but killing the others. but now i wonder, what does the comment mean " immine" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122406 (owner: 10Dzahn)
[16:50:54] <gwicke>	 akosiaris, replied at https://www.mediawiki.org/wiki/Talk:Packaging#Developer_upload
[16:51:49] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] "if nobody complained by tomorrow" [operations/dns] - 10https://gerrit.wikimedia.org/r/122719 (owner: 10Matanya)
[16:53:14] <mutante>	 greg-g: hah, same now, no SAL and forgot
[16:53:19] <mutante>	 its just automatic
[16:53:27] <greg-g>	 :)
[16:53:35] <icinga-wm>	 RECOVERY - HTTP on virt1000 is OK: HTTP OK: HTTP/1.1 302 Found - 457 bytes in 0.004 second response time  
[16:54:09] <mutante>	 that looks promising, though cert error
[16:54:12] <mutante>	 andrewbogott: 
[16:54:15] <icinga-wm>	 RECOVERY - puppetmaster https on virt1000 is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.026 second response time  
[16:56:06] <mutante>	 https://virt1000.wikimedia.org/wiki/SAL
[16:56:40] <mutante>	 yes, but getting virt1000 redirects
[16:56:49] <mutante>	 greg-g: you as well?
[16:57:05] <mutante>	 <p>The document has moved <a href="https://virt1000.wikimedia.org/">here</a>.</p>
[16:57:38] <greg-g>	 cert error.... (obviously), but it works-ish (no css/etc)
[16:57:51] <mutante>	 andrewbogott: must be a redirect somewhere
[16:58:11] <mutante>	 but yea, it did not do this with virt0 before
[16:59:04] <mutante>	 andrewbogott: mediawiki config?
[16:59:26] <mutante>	 eh, i mean, apache config of the mediawiki virtual host that is now on virt1000
[16:59:48] <mutante>	 and/or Localsettings.php 
[17:00:03] <mutante>	 with the alias for short urls
[17:07:00] <andrewbogott>	 mutante: does it look any different now?
[17:07:17] <mutante>	 andrewbogott: it does!:)
[17:07:24] <andrewbogott>	 does it look e.g. correct?
[17:07:25] <mutante>	 looks fixed
[17:07:29] <andrewbogott>	 Ok, let's see if puppet breaks it
[17:07:34] * andrewbogott  still can't see for himself :(

[17:07:48] <mutante>	 lgtm, nice
[17:08:04] <andrewbogott>	 mutante: are pages read-only?  Can you log in?
[17:08:09] <mutante>	 !log testing log to new wikitech
[17:08:10] * andrewbogott  REALLY hates not being able to see this for himself

[17:08:49] <mutante>	 andrewbogott: i can log in
[17:09:22] <andrewbogott>	 mutante: ok, so $wgReadOnly does nothing :(
[17:09:32] <mutante>	 Warning: The database has been locked for maintenance, so you will not be able to save your edits right now. You may wish to copy and paste your text into a text file and save it for later. 
[17:09:44] <mutante>	 it does 
[17:09:44] <andrewbogott>	 Ah finally my dns catches up
[17:09:49] <mutante>	 logging in wanst writing
[17:09:59] <mutante>	 trying to edit was
[17:10:21] <mutante>	 preview works, looks as designed
[17:11:33] <mutante>	 what did you do to fix it, btw
[17:12:43] <andrewbogott>	 nothing interesting, puppet is just fighting with me, wanting two competing apache confs
[17:12:46] <andrewbogott>	 still not properly fixed
[17:14:16] <grrrit-wm>	 (03PS1) 10Ottomata: Adding stat1003 and standard class in site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/122857 
[17:18:15] <grrrit-wm>	 (03PS1) 10Andrew Bogott: In production, the apache controller is always called 'wikitech' [operations/puppet] - 10https://gerrit.wikimedia.org/r/122858 
[17:18:18] <andrewbogott>	 mutante: ^
[17:18:59] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Adding stat1003 and standard class in site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/122857 (owner: 10Ottomata)
[17:19:04] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] In production, the apache controller is always called 'wikitech' [operations/puppet] - 10https://gerrit.wikimedia.org/r/122858 (owner: 10Andrew Bogott)
[17:19:38] <mutante>	 andrewbogott: missing a { 
[17:20:51] <mutante>	 manifests/openstack.pp:            require => File["/etc/apache2/sites-available/${controller_hostname}"],
[17:21:19] <mutante>	 it looks like it already did that
[17:21:33] <grrrit-wm>	 (03PS2) 10Andrew Bogott: In production, the apache controller is always called 'wikitech' [operations/puppet] - 10https://gerrit.wikimedia.org/r/122858 
[17:21:34] <mutante>	 oh, i see
[17:23:09] <grrrit-wm>	 (03CR) 10Dzahn: [C: 031] "yea, looks like on virt1000 it created virt1000.wikimedia.org in sites-available but wikitech was enabled. wondering a little why it worke" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122858 (owner: 10Andrew Bogott)
[17:23:54] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] In production, the apache controller is always called 'wikitech' [operations/puppet] - 10https://gerrit.wikimedia.org/r/122858 (owner: 10Andrew Bogott)
[17:30:31] <grrrit-wm>	 (03PS2) 10Dzahn: ms10: decom [operations/puppet] - 10https://gerrit.wikimedia.org/r/122835 (owner: 10Matanya)
[17:31:55] <grrrit-wm>	 (03CR) 10Jkrauska: "Next patch set.." (035 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/115345 (owner: 10Jkrauska)
[17:32:13] <andrewbogott>	 mutante: everything look right to you now?
[17:33:09] <grrrit-wm>	 (03PS6) 10Jkrauska: Initial commit of pmacct module and role [operations/puppet] - 10https://gerrit.wikimedia.org/r/115345 
[17:33:35] <ottomata>	 heya Reedy
[17:33:36] <ottomata>	 yt?
[17:34:08] <mutante>	 andrewbogott: https://wikitech.wikimedia.org/w/index.php?title=User:Dzahn/foobar&oldid=108279
[17:34:11] <mutante>	 yes:)
[17:34:27] <andrewbogott>	 cool
[17:34:29] <mutante>	 !log logging to eqiad wikitech after Andrew switched over
[17:34:34] <morebots>	 Logged the message, Master
[17:34:49] <grrrit-wm>	 (03PS1) 10BryanDavis: [WIP] Make role::graphite work in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/122859 
[17:34:51] <mutante>	 works
[17:39:49] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: Adding php5-pgsql to toollabs [operations/puppet] - 10https://gerrit.wikimedia.org/r/122861 
[17:43:41] <manybubbles>	 qchris: I believe my opinion on <repository> in pom mostly comes from stuff lik http://blog.sonatype.com/2009/02/why-putting-repositories-in-your-poms-is-a-bad-idea/#.Uzr6pfgzfv4
[17:43:55] <manybubbles>	 which I actually admit I haven't read all of recently, so maybe I'm out of date
[17:44:22] <ottomata>	 manybubbles:  checkin meeitng!?
[17:44:27] <ottomata>	 i'm so rornreeyy
[17:47:09] <ottomata>	 manybubbles, qchris, according to that article, i think we can't do what we want anyway
[17:47:27] <ottomata>	 we want archiva's wikimedia.mirrored repository to be as far as maven ever looks for artifacts
[17:47:30] <ottomata>	 that article says
[17:47:30] <ottomata>	 s not possible to define a mirrorOf in a POM
[17:47:38] <manybubbles>	 ottomata: sorry, was out longer then I thought.  then ^d told me that no one showed up
[17:48:20] <qchris>	 ottomata: were we gonna do mirrorOf in pom? Now I am confused.
[17:48:20] <^d>	 ottomata and I must've checked the hangout at different times :p
[17:48:45] <ottomata>	 ha, maybe
[17:48:48] <ottomata>	 think i've been there the whole time
[17:48:49] <ottomata>	 but its
[17:48:50] <ottomata>	 ok
[17:48:59] <ottomata>	 um, qchris, dunno if we need mirrorOf
[17:48:59] <ottomata>	 but
[17:49:05] <ottomata>	 we want all builds to only talk to archiva
[17:49:11] <ottomata>	 i think the only way to do that is in settings.xml, right?
[17:49:36] <grrrit-wm>	 (03CR) 10Dzahn: [C: 032] ms10: decom [operations/puppet] - 10https://gerrit.wikimedia.org/r/122835 (owner: 10Matanya)
[17:49:38] <manybubbles>	 ottomata: I'm in the hangout and you aren't there?
[17:49:58] <ottomata>	 wha?
[17:49:59] <qchris>	 ottomata: I am not sure yet.
[17:50:03] <ottomata>	 i clicked on the one from the cal
[17:50:09] <ottomata>	 https://plus.google.com/hangouts/_/wikimedia.org/search
[17:50:37] <cajoel>	 procurement requests go to ops-requests ?
[17:51:28] <ottomata>	 cajoel: 'procurement' :)
[17:51:33] <ottomata>	 queue
[17:51:49] <cajoel>	 RT queue?
[17:51:58] <ottomata>	 manybubbles: what hangout are you in? cajoel, ya
[17:52:18] <manybubbles>	 ottomata: I'm in /search
[17:52:32] <ottomata>	 oh PFFF
[17:52:33] <ottomata>	 i wasn't joined
[17:52:36] <ori>	 i'll be syncing cherry-picks of https://gerrit.wikimedia.org/r/#/c/122780/ to production in a few minutes with greg-g's approval
[17:52:36] <ottomata>	 just sitting waiting to click join
[18:02:51] <logmsgbot>	 !log ori synchronized php-1.23wmf19/includes/profiler/ProfilerSimple.php  'Iad91f1d12: Send profiled items under the correct name'
[18:02:55] <morebots>	 Logged the message, Master
[18:03:55] <logmsgbot>	 !log ori synchronized php-1.23wmf20/includes/profiler/ProfilerSimple.php  'Iad91f1d12: Send profiled items under the correct name'
[18:04:00] <morebots>	 Logged the message, Master
[18:05:01] <grrrit-wm>	 (03PS2) 10BryanDavis: [WIP] Make role::graphite work in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/122859 
[18:05:12] <hoo>	 AaronSchulz: Why do we have $wgMemCachedTimeout = 250000; which seems ridiculously high?
[18:08:52] <grrrit-wm>	 (03PS6) 10Alexandros Kosiaris: gerrit: qualify var [operations/puppet] - 10https://gerrit.wikimedia.org/r/119253 (owner: 10Matanya)
[18:10:04] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] gerrit: qualify var [operations/puppet] - 10https://gerrit.wikimedia.org/r/119253 (owner: 10Matanya)
[18:11:42] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] parsoid: logrotated file is now a parameter [operations/puppet] - 10https://gerrit.wikimedia.org/r/122564 (owner: 10Hashar)
[18:12:08] * AaronSchulz  looks at 72f1475624816b2dc9afcaf78431abc18b9ce1ca

[18:12:11] <AaronSchulz>	 maybe Reedy knows something
[18:12:52] <grrrit-wm>	 (03PS1) 10Ragesoss: Update the redirect target for education.wikimedia.org [operations/apache-config] - 10https://gerrit.wikimedia.org/r/122866 
[18:13:26] <hoo>	 AaronSchulz: Point is that springle and I got the suspicion that clients block the DB in case they wait for memcached forever
[18:14:11] <ori>	 hoo: https://twitter.com/wikimediatech/status/195729518273642496
[18:14:20] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] beta: send Parsoid log to shared dir [operations/puppet] - 10https://gerrit.wikimedia.org/r/122561 (owner: 10Hashar)
[18:16:05] <hoo>	 ori: Ok, that bug mentions 3s... but right now we have 250s
[18:17:13] <AaronSchulz>	 hoo: that's just some memcached semaphores that wait in loops+sleep. They don't distinguish "item is locked" from "can't talk to memcached". I drastically lowered the timeout for one of those in LocalFile.
[18:17:43] <AaronSchulz>	 there are other callers of BagOStuff::lock() which wait for up to 10-30sec...those should really fail faster on connection problems
[18:18:17] <AaronSchulz>	 BagOStuff really needs a clearLastError()/getLastErrorType() method or something
[18:18:32] <AaronSchulz>	 e.g. add() returning false isn't very specific ;)
[18:19:18] <hoo>	 Ok... so, what's the specific reason to have 250s in there?
[18:19:23] <mutante>	 !log ms10 - disable puppet, revoke puppet cert,salt key,icinga..
[18:19:28] <morebots>	 Logged the message, Master
[18:19:38] <hoo>	 no matter what it's waiting for, it probably shouldn't wait this long
[18:20:05] <AaronSchulz>	 it is microseconds
[18:20:20] <hoo>	 AaronSchulz: but 250k
[18:20:22] <AaronSchulz>	 $wgMemCachedTimeout that is
[18:20:30] <hoo>	 which is 250s
[18:21:27] <AaronSchulz>	 250000/1e6 ?
[18:21:39] <ori>	 hoo, 0.25
[18:21:58] <ori>	 (which may still be too high)
[18:22:35] <grrrit-wm>	 (03PS2) 10Ragesoss: Update the redirect target for education.wikimedia.org [operations/apache-config] - 10https://gerrit.wikimedia.org/r/122866 
[18:22:42] <hoo>	 you sure it's microseconds and not ms?
[18:26:33] <hoo>	 ori: Ok...         'server_failure_limit' => 1e9, this also seems quite high
[18:27:39] <ori>	 ialex: thanks again for the '-total' patch. just a short while ago i was preparing to start chasing that down and i wasn't looking forward to it. and then like magic: PATCH_FOR_REVIEW :)
[18:27:59] <mutante>	 !log ms10 - shut down
[18:28:03] <morebots>	 Logged the message, Master
[18:28:05] <Krinkle>	 manybubbles: ping
[18:28:14] <manybubbles>	 pong
[18:28:17] <manybubbles>	 Krinkle: ^^
[18:28:20] * mutante  wonders if shutting down ms13 would be political

[18:28:27] <Krinkle>	 manybubbles: Got a weird thingy with search. Getting very strange and varying results
[18:28:29] <Krinkle>	 let me give you a paste
[18:28:34] <manybubbles>	 sure
[18:28:45] <ori>	 hoo: i don't remember the exact reasons for that one either, but in general there are weird things in the memcached extension, libmemcached, and twemproxy that forced certain settings
[18:29:04] <Krinkle>	 Actually, you might want to reproduce it yourself. Log on to tin.eqiad.wmnet, and `$ mwgrep user.anonymous`
[18:29:11] <grrrit-wm>	 (03PS1) 10Ori.livneh: Tweak ScriptAliasMatch directive for perf.wm.o [operations/puppet] - 10https://gerrit.wikimedia.org/r/122868 
[18:29:14] <Krinkle>	 It searches the mediawiki namespace for a string literal on all wikis
[18:29:19] <Krinkle>	 every time I run it, the results strange dramatically
[18:29:27] <Krinkle>	 varying anywhere from 4 to 30 matches
[18:29:36] <manybubbles>	 ah, mwgrep
[18:29:54] <Krinkle>	 It seems too small for ranking to be relevant
[18:30:12] <Krinkle>	 and the pages are not actively edited, and it's not settling either. It's been randomly changing continuously, so it's not slave lag or something like that
[18:30:14] <hoo>	 ori: Well, we can't just have thousands of DB connections pile up while thousands of clients try to connect to memcached for ... time
[18:31:09] <Krinkle>	 Though it could be sync issues between slaves (how many slaves are there?), but I rarely get the same result twice. I got like pretty much every possible combination of results I can think of
[18:31:21] <Krinkle>	 between 0, 1, 14 and 30 results
[18:31:40] <Krinkle>	 manybubbles: 
[18:31:57] <manybubbles>	 Krinkle: I'm getting 20, 21, and 22 results
[18:32:26] <manybubbles>	 Krinkle: 3 slaves for most things
[18:33:09] <ialex>	 ori: even if I actually made the change that broke it? :)
[18:33:39] <grrrit-wm>	 (03PS2) 10Reedy: Non wikipedias to 1.23wmf20 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122836 
[18:34:15] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Non wikipedias to 1.23wmf20 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122836 (owner: 10Reedy)
[18:34:22] <grrrit-wm>	 (03Merged) 10jenkins-bot: Non wikipedias to 1.23wmf20 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122836 (owner: 10Reedy)
[18:36:06] <logmsgbot>	 !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: non wikipedias to 1.23wmf20
[18:36:10] <morebots>	 Logged the message, Master
[18:37:42] <logmsgbot>	 !log reedy synchronized docroot and w
[18:37:45] <morebots>	 Logged the message, Master
[18:38:07] <grrrit-wm>	 (03PS4) 10Reedy: Add zerowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119990 
[18:38:11] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] Add zerowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119990 (owner: 10Reedy)
[18:38:19] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add zerowiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119990 (owner: 10Reedy)
[18:41:17] <logmsgbot>	 !log reedy synchronized database lists files:
[18:41:21] <morebots>	 Logged the message, Master
[18:42:28] <logmsgbot>	 !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Add zerowiki
[18:42:32] <morebots>	 Logged the message, Master
[18:43:09] <logmsgbot>	 !log reedy synchronized wmf-config/InitialiseSettings.php  'touch'
[18:43:13] <morebots>	 Logged the message, Master
[18:43:36] <Reedy>	 yurik: ^^ lgtm
[18:43:42] <Reedy>	 Probably want a better logo :D
[18:43:55] <logmsgbot>	 !log reedy updated /a/common to {{Gerrit|If887effe5}}: Add zerowiki
[18:43:59] <morebots>	 Logged the message, Master
[18:44:00] <grrrit-wm>	 (03PS1) 10Reedy: zerowiki to 1.23wmf20 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122870 
[18:44:11] <manybubbles>	 Krinkle: when I run the search against elasticsearch directly I _always_ get 25 results
[18:44:27] <Krinkle>	 manybubbles: define directly?
[18:44:29] <grrrit-wm>	 (03CR) 10Reedy: [C: 032] zerowiki to 1.23wmf20 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122870 (owner: 10Reedy)
[18:44:31] <yurik>	 Reedy, awesome!!!! zero is live :)
[18:44:31] <ori>	 manybubbles: what do you mean, 'directly'?
[18:44:34] <ori>	 that
[18:44:36] <Reedy>	 Krinkle: Not indirectly
[18:44:37] <grrrit-wm>	 (03Merged) 10jenkins-bot: zerowiki to 1.23wmf20 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/122870 (owner: 10Reedy)
[18:44:48] <manybubbles>	 for i in {1..10} ; do curl -s -XPOST http://search.svc.eqiad.wmnet:9200/_all/page/_search?pretty -d @post > $i; done
[18:44:53] <Krinkle>	 Thank you for that Reedy
[18:44:58] <yurik>	 Reedy, thank you!
[18:45:10] <ori>	 ialex: yes, even if :)
[18:45:15] <logmsgbot>	 !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: zerowiki to 1.23wmf20
[18:45:15] <manybubbles>	 let me copy the python script locally and play with ti
[18:45:18] <morebots>	 Logged the message, Master
[18:46:59] <ori>	 manybubbles: add:
[18:47:10] <ori>	 print 'uri: %s' % uri
[18:47:11] <ori>	 print 'payload: %s' % json.dumps(search, sort_keys=True, indent=2)
[18:47:17] <ori>	 after line 82
[18:47:30] <ori>	 to see the request that it is generating
[18:47:49] <ori>	 you can cp `which mwgrep` to your home dir and edit it to experiment
[18:48:41] <manybubbles>	 yeah
[18:48:43] <manybubbles>	 doing it
[18:54:13] <manybubbles>	 dat timeout
[18:55:25] * ^demon|lunch  catches scrollback

[18:55:59] <Krinkle>	 ori: Is event dummy.gif supposed to result in 404?
[18:56:17] <Krinkle>	 test2.wikipedia.org, upon fiddling with VisualEditor, made this request:
[18:56:18] <Krinkle>	 https://bits.wikimedia.org/dummy.gif?%7B%22event%22%3A%7B%22points%22%3A%22ve.performance.system.activation%3D417%2Cve.performance.system.domLoad%3D265%22%7D%2C%22clientValidated%22%3Afalse%2C%22revision%22%3A7254808%2C%22schema%22%3A%22TimingData%22%2C%22webHost%22%3A%22test2.wikipedia.org%22%2C%22wiki%22%3A%22test2wiki%22%7D;
[18:56:26] <Krinkle>	 Which created a 404 Not Found in the console.
[18:56:39] <grrrit-wm>	 (03CR) 10GWicke: "It has not, but varnish has been working ok and cache purges have become very rare. We'll also move away from it hopefully soon. So I thin" [operations/puppet] - 10https://gerrit.wikimedia.org/r/72653 (owner: 10Pyoungmeister)
[18:57:30] <grrrit-wm>	 (03PS1) 10Manybubbles: mwgrep's timeout should be in seconds [operations/puppet] - 10https://gerrit.wikimedia.org/r/122871 
[18:57:40] <grrrit-wm>	 (03PS2) 10Manybubbles: mwgrep's timeout should be in seconds [operations/puppet] - 10https://gerrit.wikimedia.org/r/122871 
[18:58:07] <manybubbles>	 ori and Krinkle: ^^^^
[18:58:15] <manybubbles>	 workaround: set the timeout to really really high
[19:03:35] <ori>	 Krinkle: test2 is unique in that it is its own schema wiki and keeps its own url endpoint
[19:03:46] <ori>	 it's not actually configured to log, so a 404 is expected
[19:06:32] <Reedy>	 Anyone in ops around? Need a hanful of apaches gracefulling please... They're redirecting zerowiki to foundationwiki
[19:07:11] <Reedy>	 mw1018.eqiad.wmnet, mw1050.eqiad.wmnet, mw1061.eqiad.wmnet, mw1070.eqiad.wmnet, mw1139.eqiad.wmnet, mw1179.eqiad.wmnet
[19:08:24] <Coren>	 Reedy: Will do; gimme a min.
[19:08:37] <ori>	 Coren, Reedy: already done
[19:08:46] <Coren>	 Bah.
[19:09:28] <Reedy>	 !log ori gracefulled mw1018, mw1050, mw1061, mw1070, mw1139, mw1179
[19:09:32] <morebots>	 Logged the message, Master
[19:09:42] <Reedy>	 Great, thanks
[19:09:47] <Krinkle>	 ori: Okay, that makes sense :)
[19:10:16] <Krinkle>	 Would be nice to perhaps have it ignored differently than a 404, but at least this isn't a regression. Will ignore for now..\
[19:12:54] <grrrit-wm>	 (03PS3) 10Ori.livneh: mwgrep's timeout should be in seconds [operations/puppet] - 10https://gerrit.wikimedia.org/r/122871 (owner: 10Manybubbles)
[19:13:06] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] mwgrep's timeout should be in seconds [operations/puppet] - 10https://gerrit.wikimedia.org/r/122871 (owner: 10Manybubbles)
[19:13:46] <grrrit-wm>	 (03CR) 10Manybubbles: "That looks better." [operations/puppet] - 10https://gerrit.wikimedia.org/r/122871 (owner: 10Manybubbles)
[19:14:06] <ori>	 manybubbles: does 30s make sense though, as a value?
[19:14:14] <manybubbles>	 sure
[19:14:17] <manybubbles>	 its a long time
[19:14:20] <manybubbles>	 but that should be ok
[19:14:29] * ori  nods.

[19:15:18] <manybubbles>	 the requests looked like they were takinga  few seconds 
[19:15:27] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032 V: 032] mwgrep's timeout should be in seconds [operations/puppet] - 10https://gerrit.wikimedia.org/r/122871 (owner: 10Manybubbles)
[19:19:55] <ori>	 manybubbles, Krinkle: I updated tin and the patch seems to have fixed it
[19:19:57] <ori>	 watch '{ mwgrep user.anonymous | tail -1 ; }'
[19:20:03] <ori>	 consistently 22 results
[19:20:06] <manybubbles>	 yay
[19:20:11] <ori>	 thanks very much
[19:20:17] <manybubbles>	 30ms is very fast for a timeout
[19:20:23] <manybubbles>	 so fast it was probably not really respecting it
[19:20:25] <ori>	 yeah, that's actually pretty cool
[19:20:58] <manybubbles>	 it checks it at various points in the process and if it goes over it gives up and returns what it can
[19:21:13] <manybubbles>	 like, after each hit it identifies, for example
[19:39:02] <grrrit-wm>	 (03CR) 10Tim Landscheidt: [C: 031] Adding php5-pgsql to toollabs [operations/puppet] - 10https://gerrit.wikimedia.org/r/122861 (owner: 10Alexandros Kosiaris)
[19:40:26] <icinga-wm>	 PROBLEM - Puppet freshness on virt0 is CRITICAL: Last successful Puppet run was Tue 01 Apr 2014 04:39:53 PM UTC  
[19:40:53] <Nemo_bis>	 scfc_de: how does one install php 5.4+ on a labs instance?
[19:45:28] <scfc_de>	 Nemo_bis: AFAICS, 5.4 is not part of Ubuntu Precise?  Then you'll need to backport/download backports, put them in /data/project/repo and enable labsdebrepo.  In other words: If your project works reasonably with 5.3, I would stay with that :-).
[19:46:41] <matanya>	 greg-g: what happened to the beta cluster docs ?
[19:47:48] <greg-g>	 matanya: which?
[19:47:51] <Nemo_bis>	 scfc_de: it doesn't; I'll run it on my home machine, thanks for the explanation
[19:48:03] <matanya>	 found, on wikitech. the link broke. fixing
[19:48:18] <greg-g>	 ty
[19:48:22] <greg-g>	 (weird)
[19:48:43] <greg-g>	 matanya: https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep ?
[19:48:48] <matanya>	 yes
[19:48:56] * greg-g  nods

[19:52:48] <matanya>	 ok, found the culprit greg-g : https://www.mediawiki.org/wiki/Beta_cluster
[19:53:06] <matanya>	 it is a local link instead of cross site link
[19:53:39] <greg-g>	 was*?
[19:53:41] <greg-g>	 wfm
[19:54:00] <matanya>	 see my diff greg-g 
[19:54:15] <greg-g>	 good point :P
[19:54:33] <greg-g>	 I wonder why that didn't work...
[19:55:25] <matanya>	 sorry to bug you for such a thing, thought it was related to wikitech take down earlier
[19:55:26] <greg-g>	 matanya: there, made it work with sitelinks
[19:55:29] * greg-g  nods

[19:55:32] <greg-g>	 no worries
[19:59:36] <icinga-wm>	 ACKNOWLEDGEMENT - Puppet freshness on virt0 is CRITICAL: Last successful Puppet run was Tue 01 Apr 2014 04:39:53 PM UTC andrew bogott Virt0 is on the way out!
[20:06:34] <thedj>	 i have alot of trouble today with gerrit and bugzilla
[20:07:03] <thedj>	 My connection seems spotty if best and http://pastebin.com/wt0nj7uY tells me i'm being routed trough ntt.net
[20:07:23] <thedj>	 which is strange for my amsterdam connection.
[20:07:44] <matanya>	 great summary regarding emery jgage 
[20:08:00] <thedj>	 anyone from europe got similar troubles ?
[20:09:20] <grrrit-wm>	 (03CR) 10Rschen7754: [C: 031] Add importsources to enwikivoyage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121090 (owner: 10John F. Lewis)
[20:15:12] <grrrit-wm>	 (03CR) 10Hashar: "See also Bug 63362 - ganglia down / needs reinstall" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122790 (owner: 10Hashar)
[20:16:08] <grrrit-wm>	 (03PS1) 10Hashar: ganglia: fix some missing paths for labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/123040 
[20:25:51] <grrrit-wm>	 (03PS3) 10BryanDavis: [WIP] Make role::graphite work in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/122859 
[20:30:38] <AaronSchulz>	 04:22 logmsgbot_: tstarling synchronized wmf-config/mc.php 'increased wgMemCachedTimeout from 500ms to 3000ms for bug 35900' 
[20:31:04] <AaronSchulz>	 heh, we don't even use mc for sessions anymore
[20:33:36] <AaronSchulz>	 250ms isn't so bad by comparison :)
[20:39:27] <grrrit-wm>	 (03PS2) 10Tim Landscheidt: Tools: Install php5-pgsql [operations/puppet] - 10https://gerrit.wikimedia.org/r/122861 (owner: 10Alexandros Kosiaris)
[20:39:45] <grrrit-wm>	 (03CR) 10Tim Landscheidt: [C: 031] Tools: Install php5-pgsql [operations/puppet] - 10https://gerrit.wikimedia.org/r/122861 (owner: 10Alexandros Kosiaris)
[20:42:51] <matanya>	 apergos: still awake ?
[20:45:04] <matanya>	 Jeff_Green: as the duty hero, can you please look at https://rt.wikimedia.org/Ticket/Display.html?id=6352 and check if ms7/8 are in fact shut down?
[20:45:18] <Jeff_Green>	 yup, sec
[20:45:53] <Jeff_Green>	 solaris!?
[20:46:10] <matanya>	 :)
[20:46:11] <matanya>	 yes
[20:46:27] <Jeff_Green>	 i have a feeling this is going to require onsite attention
[20:47:07] <matanya>	 i hope not, no ping?
[20:47:44] <matanya>	 ah, not in dns at all
[20:47:48] <Jeff_Green>	 matanya: right
[20:47:57] <Jeff_Green>	 and who knows what the console situation is
[20:48:00] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Create "/a/backup/public" on wikitech. [operations/puppet] - 10https://gerrit.wikimedia.org/r/123043 
[20:48:06] <matanya>	 Jeff_Green: is it in racktables?
[20:48:12] <grrrit-wm>	 (03PS1) 10Hashar: ganglia: graphdir must be an absolute path [operations/puppet] - 10https://gerrit.wikimedia.org/r/123044 
[20:49:20] <Jeff_Green>	 matanya: the lights-out mgmt IP for ms7 does not respond
[20:49:30] <matanya>	 RobH might know
[20:49:57] <matanya>	 and if he doesn't Chris Johnson might. if he doesn't as well - on site
[20:50:19] <matanya>	 thanks for checking Jeff_Green 
[20:50:38] <Jeff_Green>	 yeah, no eth or lights-out answers for either box
[20:52:33] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Create "/a/backup/public" on wikitech. [operations/puppet] - 10https://gerrit.wikimedia.org/r/123043 (owner: 10Andrew Bogott)
[20:53:03] <grrrit-wm>	 (03CR) 10coren: [C: 032] "Trivial package addition." [operations/puppet] - 10https://gerrit.wikimedia.org/r/122861 (owner: 10Alexandros Kosiaris)
[21:00:17] <andrewbogott>	 I like how wikitech-static says "Number of users:  1"
[21:04:03] <Damianz>	 Should make it 0 and troll updates
[21:08:40] <YuviPanda>	 Coren: small patch coming up
[21:08:42] <grrrit-wm>	 (03PS1) 10Yuvipanda: dynamicproxy: Add explicit DNS resolver to tools proxy role [operations/puppet] - 10https://gerrit.wikimedia.org/r/123088 
[21:08:45] <YuviPanda>	 Coren: ^
[21:08:52] <YuviPanda>	 preventing puppet run now... :)
[21:09:25] <grrrit-wm>	 (03CR) 10coren: [C: 032] "The change, she is tiny!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/123088 (owner: 10Yuvipanda)
[21:09:47] <Coren>	 Will merge as soon as jenkins approves.  :-)
[21:09:52] <YuviPanda>	 Coren: ah, right.
[21:10:09] <YuviPanda>	 Coren: jenkins has been slowish today, I think.
[21:10:29] <YuviPanda>	 Coren: I'm going to look into proxylistener in the meantime. Do you have time to give a hand with the upstart job?
[21:11:11] <Coren>	 YuviPanda: Sure.  That thing doesn't daemonize right?
[21:11:19] <YuviPanda>	 Coren: nope.  
[21:11:53] <YuviPanda>	 Coren: it's also a threaded server. I can trivially make it a forking one with a simple code change. Would that affect upstart?
[21:12:15] <Coren>	 Yes, and it's better if it isn't.
[21:12:41] <YuviPanda>	 Coren: ok, I'll leave it to be threaded :) Theoretically it should give us less performance, but we'll see. There are plenty f other avenues for optimization :)
[21:12:41] <Coren>	 Wait, wait.  proxylistener runs exactly once on the proxy?
[21:12:48] <YuviPanda>	 Coren: yes, exactly once.
[21:13:31] <Coren>	 Oh!  We can't do the portgranter trick then; /that/ is meant to run one instance per node and connects through a unix socket.
[21:13:41] <YuviPanda>	 Coren: right now it spawns a new thread for each incoming routing-request, holds on to the open socket, and cleans up the routing-request when socket closes. I can make it spawn a new process instead too.
[21:13:43] <YuviPanda>	 Coren: oh, why not?
[21:13:55] <Coren>	 ... and that explains why you do identd.  :-)
[21:13:59] <YuviPanda>	 Coren: we can verify with  identd
[21:14:00] <YuviPanda>	 yeayh
[21:14:01] <YuviPanda>	 :)
[21:14:30] <YuviPanda>	 since we (theoretically :P) have only trusted admins who won't go 'identd is stupid, we should rewrite it so we can spoof cool names on IRC!'
[21:14:36] <Coren>	 Don't mess with proxylistener then, I'll modify portgranter.
[21:14:45] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] ganglia: fix some missing paths for labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/123040 (owner: 10Hashar)
[21:14:52] <YuviPanda>	 Coren: oh. wheee! :)
[21:15:15] <Coren>	 What does proxylistener expect on the socket?
[21:15:45] <YuviPanda>	 Coren: moment
[21:17:23] <YuviPanda>	 Coren: <regex-to-match-url-against>\n<host-to-route-to>:<port>
[21:17:50] <Coren>	 Does it then return anything or do I just hold the socket open?
[21:18:08] <YuviPanda>	 Coren: just hold the socket open.
[21:18:13] <YuviPanda>	 Coren: it keeps the route alive as long as socket is open
[21:18:27] * Coren  nods. Like portgrabber.

[21:18:31] <YuviPanda>	 am documenting this now
[21:18:38] <YuviPanda>	 Coren: yeah, I read the code before writing this :)
[21:19:28] <Coren>	 Wait, you allow regexes?  How do you actualy validate that against the username?
[21:19:48] <YuviPanda>	 Coren: prefixed with username
[21:20:09] <YuviPanda>	 Coren: forgot to mention that. sorry :)
[21:20:11] <Coren>	 What about /csbot|/evil
[21:20:13] <Coren>	 :-)
[21:20:41] <YuviPanda>	 hmm, right. so I need to validate username separately and regex separately.
[21:20:51] <grrrit-wm>	 (03PS2) 10Hashar: ganglia: graphdir must be an absolute path [operations/puppet] - 10https://gerrit.wikimedia.org/r/123044 
[21:21:10] <Coren>	 I'm thinging that "prefix" is just as useful as regex, and considerably less full of holes and user confusion.  :-)
[21:21:16] <YuviPanda>	 Coren: oh yeah, that's what I am doing :)
[21:21:24] <YuviPanda>	 Coren: as in, I am validating username and regex separately
[21:21:39] <Coren>	 Ah.  So <prefix-to-match-url-against>\n<host>:<port>  :-)
[21:22:07] <YuviPanda>	 Coren: yeah, I'm reading the code to refamiliarize myself. Apparently me from a few months ago did think of these :D
[21:22:13] <YuviPanda>	 Coren: not really
[21:22:16] <YuviPanda>	 Coren: it is regex
[21:22:26] <Coren>	 It shouldn't be.  That's faught with pitfalls.
[21:22:34] <YuviPanda>	 Coren: <regex-to-match-part-of-url-after-toolname-against>\n<host>:<port>
[21:22:46] <Coren>	 Hmmm.  I guess that's better.
[21:22:52] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] ganglia: graphdir must be an absolute path [operations/puppet] - 10https://gerrit.wikimedia.org/r/123044 (owner: 10Hashar)
[21:23:02] <Coren>	 But in practice, that'll never be used by portgrabber.
[21:23:08] <Coren>	 Meh.
[21:23:25] <YuviPanda>	 :)
[21:24:34] <grrrit-wm>	 (03PS3) 10Hashar: ganglia: graphdir must be an absolute path [operations/puppet] - 10https://gerrit.wikimedia.org/r/123044 
[21:26:17] <Coren>	 Hah.  Ima just /add/ the connect to portgrabber so that both are maintained.
[21:27:12] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] ganglia: graphdir must be an absolute path [operations/puppet] - 10https://gerrit.wikimedia.org/r/123044 (owner: 10Hashar)
[21:27:57] <hashar>	 !log jenkins killed stuck build (5 hours+) of beta-update-databases-eqiad . Might have been blocking Jenkins build queue
[21:28:01] <morebots>	 Logged the message, Master
[21:28:22] <Coren>	 YuviPanda: What is the port?
[21:28:57] <YuviPanda>	 Coren: :)
[21:28:58] <YuviPanda>	 Coren: 8282
[21:31:38] <grrrit-wm>	 (03PS4) 10BryanDavis: [WIP] Make role::graphite work in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/122859 
[21:32:31] <Coren>	 YuviPanda: So, "/.*\nsomehost:1234\n" would work?
[21:32:39] <YuviPanda>	 yes
[21:34:07] <grrrit-wm>	 (03PS1) 10coren: Tool Labs: make portgrabber talk to the YuviProxy [operations/puppet] - 10https://gerrit.wikimedia.org/r/123097 
[21:34:16] <YuviPanda>	 Coren: you can run proxylistener locally to test. It only requires that you have redis installed locally (+ pyredis), no other dependencies.
[21:34:20] <Coren>	 ^^ not to merge before the proxy is ready, obviously.  :-)
[21:34:51] <YuviPanda>	 heh
[21:35:12] <YuviPanda>	 hmm, my perl skills are zero so I've no idea what it does
[21:35:21] <YuviPanda>	 Coren: that patch you +2'd is still unmerged :(
[21:35:37] <grrrit-wm>	 (03PS2) 10coren: Tool Labs: make portgrabber talk to the YuviProxy [operations/puppet] - 10https://gerrit.wikimedia.org/r/123097 
[21:35:55] * Coren  makes the connection to yuviproxy optional, actually.

[21:38:05] <YuviPanda>	 Coren: how does it know what port to use locally?
[21:38:24] <Coren>	 portgranter will still be the one to give it.
[21:38:41] <Coren>	 And the patch is merged.
[21:38:50] <Coren>	 BTW.  So I don't know why you think it isn't.
[21:38:51] <YuviPanda>	 Coren: oh, the *current* portgranter?
[21:38:55] * Coren  nods.

[21:39:04] <Coren>	 Ima keep both, at least for the forseeable future.
[21:39:13] <Coren>	 portgrabber just keeps both sockets open.
[21:39:22] <YuviPanda>	 Coren: nice solution! :)
[21:39:26] <YuviPanda>	 I hadn't thought of that
[21:39:33] <YuviPanda>	 although it is duplication and adds complexity
[21:39:53] <YuviPanda>	 Coren: I can make proxylistener allocate ports fairly easily if you want, but yeah good to get it started this way
[21:40:13] <Coren>	 This way I keep the scoreboard; which is also a good thing.
[21:40:17] <YuviPanda>	 Coren: :)
[21:40:19] <YuviPanda>	 Coren: cool.
[21:40:31] <YuviPanda>	 Coren: let me write the upstart job
[21:41:32] <YuviPanda>	 Coren: ok, it needs nginx newer version. deb needs to be copied over from proxy-project.
[21:42:30] <hashar>	 !log Ganglia in labs is more or less back in activity: http://ganglia.wmflabs.org/   No clue what it is graphing though
[21:42:35] <morebots>	 Logged the message, Master
[21:43:38] <grrrit-wm>	 (03PS1) 10Yuvipanda: toollabs: Install python-redis on dynamicproxy host [operations/puppet] - 10https://gerrit.wikimedia.org/r/123104 
[21:43:39] <YuviPanda>	 Coren: ^
[21:44:27] <Coren>	 YuviPanda: Put it in /data/project/.system/deb/ that makes it available to tools instances.
[21:44:36] <YuviPanda>	 Coren: yeah, I'll copy it over in a bit.
[21:45:33] <grrrit-wm>	 (03CR) 10coren: "One inline comment." (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/123104 (owner: 10Yuvipanda)
[21:46:34] <YuviPanda>	 Coren: didn't realize we had custom deb for python-redis?
[21:46:39] <grrrit-wm>	 (03CR) 10Yuvipanda: toollabs: Install python-redis on dynamicproxy host (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/123104 (owner: 10Yuvipanda)
[21:47:13] <Coren>	 I don't think we do, but I keep labs at ensure => latest because we have several packages which we do keep as backports, etc.
[21:47:43] <Coren>	 ensure => present is good for prod predictability, but can lead to not all instances having the same version of packages which is often disastrous in grids.
[21:47:55] <YuviPanda>	 Coren: ah, ok :) cool
[21:48:17] <grrrit-wm>	 (03PS2) 10Yuvipanda: toollabs: Install python-redis on dynamicproxy host [operations/puppet] - 10https://gerrit.wikimedia.org/r/123104 
[21:48:21] <YuviPanda>	 there
[21:51:56] <YuviPanda>	 Coren: ^
[21:52:10] <YuviPanda>	 Coren: also, _all packages go into debs/all and amd64 into debs/amd64 right?
[21:52:12] <grrrit-wm>	 (03CR) 10coren: [C: 032] toollabs: Install python-redis on dynamicproxy host [operations/puppet] - 10https://gerrit.wikimedia.org/r/123104 (owner: 10Yuvipanda)
[21:52:24] <Coren>	 YuviPanda: Predictably enough.  :-)
[21:52:28] <YuviPanda>	 Coren: :)
[21:52:30] <YuviPanda>	 just making sure
[21:52:36] * YuviPanda  copies just the minimum number of packages

[21:58:05] <YuviPanda>	 Coren: cool, appropriate nginx version installed :)
[21:58:12] <YuviPanda>	 now I just need upstart job
[22:02:45] <grrrit-wm>	 (03CR) 10coren: [C: 032] "I agree with myself." [operations/puppet] - 10https://gerrit.wikimedia.org/r/123097 (owner: 10coren)
[22:03:23] <Coren>	 YuviPanda: Hypothetically, portgrabber will now also notify the proxy if it is available.
[22:03:33] <YuviPanda>	 Coren: sweeet! :)
[22:03:49] <Coren>	 Ah, my food is ready.
[22:05:16] <YuviPanda>	 Coren: ^
[22:05:18] <grrrit-wm>	 (03PS1) 10Yuvipanda: toollabs: Setup proxylistener.py on proxy host [operations/puppet] - 10https://gerrit.wikimedia.org/r/123112 
[22:05:19] <YuviPanda>	 err
[22:05:20] <YuviPanda>	 Coren: ^
[22:07:27] <YuviPanda>	 Coren: unfortunately it's about to hit 4AM, so I should sleep now :( I can continue again tomorrow.
[22:11:06] <grrrit-wm>	 (03PS5) 10BryanDavis: [WIP] Make role::graphite work in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/122859 
[22:12:16] <YuviPanda>	 Coren: I'm going to stick around for some more time (10-15m) doing some apps work before I hit the bed
[22:13:18] * Reedy  gives YuviPanda a stick

[22:13:31] * YuviPanda  sticks Reedy to a boat

[22:28:19] <YuviPanda>	 Coren: ok, off for real now. night!
[23:10:20] <ori>	 OK, ld time
[23:10:22] <ori>	 or swat
[23:10:31] <ebernhardson>	 swat now i suppose :)
[23:12:46] <ori>	 yep
[23:12:49] <ori>	 who's doing it?
[23:12:59] <ori>	 i don't really feel like it; are you up for it?
[23:13:03] <ebernhardson>	 sure
[23:13:05] <ori>	 if not, i can do it
[23:13:08] <ori>	 ah cool, thanks
[23:14:11] <ebernhardson>	 ori: hmm, these are both to master but no mention of destination branches in deployments page.  Do you normally do wmf19 & wmf20?
[23:14:43] * jamesofur  doesn't think his really matters branch wise...

[23:15:37] <ebernhardson>	 jamesofur: oh your right :)
[23:16:05] <grrrit-wm>	 (03CR) 10EBernhardson: [C: 032] Add config file for bing webmaster tools. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121297 (owner: 10Jalexander)
[23:16:22] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add config file for bing webmaster tools. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121297 (owner: 10Jalexander)
[23:17:31] <jamesofur>	 <3
[23:17:59] <ori>	 ebernhardson: that's for both 19 and 20 -- want me to bump the submodule?
[23:18:32] <ebernhardson>	 ori: i can do that, its easy enough
[23:19:23] <ebernhardson>	 although, for some reason it seems my ssh agent isn't forwarding into gerrit ... sec
[23:21:36] <icinga-wm>	 PROBLEM - MySQL InnoDB on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[23:22:26] <icinga-wm>	 RECOVERY - MySQL InnoDB on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds  
[23:22:56] <icinga-wm>	 PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[23:22:56] <icinga-wm>	 PROBLEM - MySQL Recent Restart on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[23:23:45] <ebernhardson>	 PHP Parse error:  syntax error, unexpected T_STRING in /a/common/docroot/wikipedia.org/BingSiteAuth.xml on line 1
[23:23:52] <ebernhardson>	 something in sync-file is trying to parse that as php, 
[23:24:12] <jamesofur>	 hmm
[23:24:31] <jamesofur>	 that's odd ... [unsurprisingly 'yes' it does have php syntax issues]
[23:24:56] <icinga-wm>	 RECOVERY - MySQL Recent Restart on db1047 is OK: OK 605543 seconds since restart  
[23:24:57] <grrrit-wm>	 (03CR) 10Krinkle: Use the BetaFeatures whitelist for production to avoid accidental deploys (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121892 (owner: 10Jforrester)
[23:25:46] <icinga-wm>	 RECOVERY - MySQL Idle Transactions on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds  
[23:28:29] <jamesofur>	 hmm, yeah ,looking at the script it appears to php lint check every file...
[23:29:53] <ebernhardson>	 hmm, sync-dir looks like it would do the same
[23:30:56] <icinga-wm>	 PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[23:31:13] <Krinkle>	 ebernhardson: jamesofur: We don't check whether the file is .php because there are quite a few different extensions that we run php on, and then there is php being able to "include" arbitrary files (regardless of the file extension), like .inc. This works in practice because every file is valid php (the engine only starts at <?php, anything else is passed through transparently)
[23:31:19] <Krinkle>	 I guess it breaks because it starts with <?
[23:31:37] <Krinkle>	 so if this file were to be included by php, or executed by apache as php (the latter is not possible in our config), it would indeed fail.
[23:31:48] <jamesofur>	 ebernhardson: sync-docroot?
[23:32:06] <Krinkle>	 We could either exclude .xml from that check, or disable <? syntax (once we're sure we no longer use it anywhere)
[23:32:17] <ebernhardson>	 jamesofur: hmm, maybe sec
[23:32:52] <logmsgbot>	 !log ebernhardson synchronized docroot and w
[23:32:56] <icinga-wm>	 RECOVERY - MySQL Idle Transactions on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds  
[23:32:58] <morebots>	 Logged the message, Master
[23:32:59] <ebernhardson>	 jamesofur: perfect!
[23:33:00] <ebernhardson>	 thanks
[23:33:04] <jamesofur>	 thank you!
[23:33:16] <ori>	 ebernhardson: I did the submodule bumps meanwhile
[23:33:21] <ori>	 for wikimediaevents, i mean
[23:34:51] <ori>	 ebernhardson: nm, your cherry picks are fine. thanks. i plopped an eventlogging extension update on there too.
[23:36:31] <ebernhardson>	 oh, well if you already updated the other part i guess thats easier :) will undo the cherry picks
[23:36:43] * ebernhardson  should watch the other channels

[23:38:05] <grrrit-wm>	 (03CR) 10Jforrester: Use the BetaFeatures whitelist for production to avoid accidental deploys (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121892 (owner: 10Jforrester)
[23:40:38] <ori>	 ebernhardson: don't undo the cherry-picks; how you did it is fine
[23:41:04] <ori>	 actually, since you don't have the submodule updates, let's go with my two patches. but you still don't need to revert yours.
[23:46:39] <logmsgbot>	 !log ebernhardson synchronized php-1.23wmf20/extensions/WikimediaEvents/  'Update WikimediaEvents to master'
[23:46:43] <morebots>	 Logged the message, Master
[23:47:05] <logmsgbot>	 !log ebernhardson synchronized php-1.23wmf20/extensions/EventLogging/  'Update EventLogging to master'
[23:47:09] <morebots>	 Logged the message, Master
[23:48:29] <logmsgbot>	 !log ebernhardson synchronized php-1.23wmf19/extensions/EventLogging/  'Update EventLogging to master'
[23:48:33] <morebots>	 Logged the message, Master
[23:48:39] <ori>	 ebernhardson: thanks!
[23:48:43] <logmsgbot>	 !log ebernhardson synchronized php-1.23wmf19/extensions/WikimediaEvents/  'Update WikimediaEvents to master'
[23:48:48] <morebots>	 Logged the message, Master