[00:02:12] (03PS1) 10Ryan Lane: Revert "Temporarily move OC traffic to eqiad" [operations/dns] - 10https://gerrit.wikimedia.org/r/91098 [00:02:35] Ryan_Lane: does it work? [00:03:00] let me do one last set of tests before I say yes ;) [00:03:04] I wasn't merging that just yet [00:03:10] but getting ready for it [00:03:12] k [00:03:15] I'm sort of around [00:04:23] <^demon|lunch> Ouch. We do not do bengali better. [00:08:53] looks like it's working properly to me [00:09:42] (03CR) 10Ryan Lane: [C: 032] Revert "Temporarily move OC traffic to eqiad" [operations/dns] - 10https://gerrit.wikimedia.org/r/91098 (owner: 10Ryan Lane) [00:27:41] (03CR) 10Awjrichards: "Alternatively, we could keep the www. rule and I can add a separate rule specifically for the m. domain." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91058 (owner: 10Awjrichards) [00:29:09] (03CR) 10Ryan Lane: [C: 032] Salt 0.17 compatibility for deployment scripts [operations/puppet] - 10https://gerrit.wikimedia.org/r/89495 (owner: 10Ryan Lane) [00:38:05] (03PS1) 10Springle: depool db1018 while cloning [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91102 [00:38:26] (03PS1) 10Ryan Lane: Don't reference pillars before assigned [operations/puppet] - 10https://gerrit.wikimedia.org/r/91103 [00:39:01] (03CR) 10Springle: [C: 032] depool db1018 while cloning [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91102 (owner: 10Springle) [00:40:18] !log springle synchronized wmf-config/db-eqiad.php 'depool db1018 while cloning' [00:40:33] Logged the message, Master [00:42:34] (03PS1) 10Springle: db1034 using mariadb [operations/puppet] - 10https://gerrit.wikimedia.org/r/91104 [00:43:48] (03CR) 10Springle: [C: 032] db1034 using mariadb [operations/puppet] - 10https://gerrit.wikimedia.org/r/91104 (owner: 10Springle) [00:45:32] springle, have time today to work on getting https://gerrit.wikimedia.org/r/#/c/88666/ merged? [00:46:42] (03CR) 10Ryan Lane: [C: 032] Don't reference pillars before assigned [operations/puppet] - 10https://gerrit.wikimedia.org/r/91103 (owner: 10Ryan Lane) [00:48:37] andrewbogott: certainly [00:49:08] It had conflicts with a recent patch of yours, so I'd appreciate you verifying that I didn't break any of that... [00:49:13] Then we can merge and watch it go :) [00:50:19] conflicts are hard to track when a patch moves a file :( git just flagged about 150 lines as conflicting [00:52:13] andrewbogott: go ahead [00:52:39] (03PS1) 10Springle: depool db1007 while cloning [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91105 [00:53:01] (03CR) 10Springle: [C: 032] depool db1007 while cloning [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91105 (owner: 10Springle) [00:53:35] !log springle synchronized wmf-config/db-eqiad.php 'depool db1007 while cloning' [00:53:47] Logged the message, Master [00:53:58] (03CR) 10Andrew Bogott: [C: 032] Move mysql_wmf into a module. [operations/puppet] - 10https://gerrit.wikimedia.org/r/88666 (owner: 10Andrew Bogott) [00:56:01] ok, watching a puppet run on db1044 [00:57:03] PROBLEM - mysqld processes on db1041 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [00:57:23] springle, here's the diff: https://dpaste.de/Iy2o [00:57:31] Not urgent but probably still a mistake of some sort... [00:58:14] Who do I tase to start a new mailing list? I tried https://bugzilla.wikimedia.org/show_bug.cgi?id=55635, but this is a rumoured black hole. [00:58:50] awight: Some are managed by ops and some are managed by OIT. And I know not the difference. [00:58:55] As an Op, I recommend you contact OIT :) [00:58:58] <^d> awight: That's the right place. [01:00:15] niiice. OK thanks, I'll rattle some bones. [01:02:16] andrewbogott: fairly painless. good job. all that an only a file perms tweak needed :) [01:02:21] so far... ;) [01:03:25] weirdly, permissions were not specified either in the before or after [01:03:41] Suppose the permissions of the files /in git/ matter? [01:04:42] <^d> You can check in permissions in git. [01:04:42] and who the heck is user 999? [01:04:53] You can, but would puppet maintain them when installing a file? [01:04:58] Surely it applies defaults instead [01:05:05] <^d> Yeah it should. [01:05:45] <^d> I consider checking permissions into git to be a helpful thing to provide mostly-sane defaults for people. [01:05:58] <^d> But you shouldn't rely on them and should set permissions appropriately on things :) [01:06:02] an explicit owner=> group=> mode=> in the file block in puppet safest? [01:06:16] Yeah, that's surely safest. I just wonder why it changed, in this case... [01:06:20] <^d> Safest. Certainly can't hurt. [01:06:20] anyway I'll just specify in puppet [01:06:45] * andrewbogott curses [01:06:54] there are three files specified in that puppet block but only two got their perms changed [01:06:57] !log upgrading db1034 & db1041 to precise [01:07:12] Logged the message, Master [01:11:02] (03PS1) 10Andrew Bogott: Specify group & permissions for a couple of files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/91112 [01:11:42] springle, ^ [01:13:33] PROBLEM - DPKG on db1041 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [01:13:57] andrewbogott: they don't need owner=> ? db1044 has both owner and group 999 [01:14:29] Hm, I was only responding to the diff when puppet applied. [01:14:41] I don't actually know what the owner/group/permissions /should/ be. [01:14:43] Do you have an opinion? [01:14:56] i think they should be root:root 644 [01:15:03] PROBLEM - MySQL Processlist on db1041 is CRITICAL: Connection refused by host [01:15:04] for all three of those files [01:15:23] ok [01:15:53] PROBLEM - MySQL InnoDB on db1041 is CRITICAL: Connection refused by host [01:15:57] we can only break some monitoring here :) [01:16:31] host db1041 [01:16:38] shutup [01:17:01] (03PS2) 10Andrew Bogott: Specify group & permissions for a couple of files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/91112 [01:18:59] springle: ^ ? [01:19:49] go for it [01:20:12] (03CR) 10Andrew Bogott: [C: 032] Specify group & permissions for a couple of files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/91112 (owner: 10Andrew Bogott) [01:21:57] hm… ok, looks reasonable on my end. [01:22:14] yep [01:23:49] ok -- I'm going to step away in a few minutes, but please email me if you see anything misbehaving. I'll be up for a while yet. [01:24:16] andrewbogott: an doing a couple upgrade/installs now. will watch for discrepancies [01:24:26] thx [01:26:31] !log start xtrabackup clone db1018 to db1034 [01:26:47] Logged the message, Master [01:31:05] (03PS1) 10Springle: db1041 using mariadb [operations/puppet] - 10https://gerrit.wikimedia.org/r/91113 [01:32:05] (03CR) 10Springle: [C: 032] db1041 using mariadb [operations/puppet] - 10https://gerrit.wikimedia.org/r/91113 (owner: 10Springle) [01:37:13] PROBLEM - mysqld processes on db1007 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [01:38:13] RECOVERY - mysqld processes on db1007 is OK: PROCS OK: 1 process with command name mysqld [01:40:59] !log start xtrabackup clone db1007 to db1041 [01:41:15] Logged the message, Master [02:10:13] (03CR) 10Fabriceflorin: [C: 031] "Sounds good to me." [operations/puppet] - 10https://gerrit.wikimedia.org/r/91084 (owner: 10Dzahn) [02:25:12] !log LocalisationUpdate completed (1.22wmf22) at Tue Oct 22 02:25:12 UTC 2013 [02:25:29] Logged the message, Master [02:39:07] Would appreciate if any ops people could weigh in on https://bugzilla.wikimedia.org/show_bug.cgi?id=55981 [02:39:15] Trying to figure out a better Redis configuration for MediaWiki-Vagrant. [02:39:32] !log LocalisationUpdate completed (1.22wmf21) at Tue Oct 22 02:39:32 UTC 2013 [02:39:45] Logged the message, Master [03:00:25] !log upgrading db1017 to precise [03:00:39] Logged the message, Master [03:04:58] springle: Let's say someone wanted to do an ALTER on the revision table of Wikimedia wikis. Doable? How painful? [03:05:49] revision has a PK so it can be done online. load spike, but possible [03:06:10] https://bugzilla.wikimedia.org/show_bug.cgi?id=55398#c4 [03:06:19] Does database size matter? [03:06:32] I mean, is there enough space to add a column without overflowing disks? [03:06:46] !log LocalisationUpdate ResourceLoader cache refresh completed at Tue Oct 22 03:06:46 UTC 2013 [03:06:46] Or does anyone need to be worried about that kind of thing. revision is big... [03:06:57] database table size * [03:07:01] Logged the message, Master [03:10:07] we have space for that alter [03:10:35] Okay, thanks. :-) That'll help push the bug forward a bit, I think. [03:11:29] Long-term, adding this column should reduce database load, I think. Maybe. Rather than moving rows between tables, we'll be able to do simple updates on a bit field, as I understand it. [03:12:01] i'll comment on the bug [03:12:16] That'd be awesome. [03:30:58] (03CR) 10Dzahn: "so i suppose /etc/ssl/certs/ca-certificates.crt must be included in the .chained.pem or specified additionally or this needs to keep SSLCA" [operations/puppet] - 10https://gerrit.wikimedia.org/r/90676 (owner: 10Dzahn) [03:45:30] (03PS1) 10Springle: repool db1007, warm up db1041 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91121 [03:46:01] (03CR) 10Springle: [C: 032] repool db1007, warm up db1041 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91121 (owner: 10Springle) [03:46:50] !log springle synchronized wmf-config/db-eqiad.php 'repool db1007, warm up db1041' [03:47:05] Logged the message, Master [04:12:19] (03PS1) 10Dzahn: remove nomcom wiki Apache config [operations/apache-config] - 10https://gerrit.wikimedia.org/r/91124 [04:20:15] (03CR) 10Dzahn: "no db: Unknown database 'nomcomwiki'." [operations/apache-config] - 10https://gerrit.wikimedia.org/r/91124 (owner: 10Dzahn) [04:27:47] (03PS1) 10Dzahn: remove nomcom wiki [operations/dns] - 10https://gerrit.wikimedia.org/r/91125 [04:40:13] RECOVERY - search indices - check lucene status page on search18 is OK: HTTP OK: HTTP/1.1 200 OK - 55880 bytes in 0.112 second response time [05:09:37] (03CR) 10Ori.livneh: "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/90733 (owner: 10Physikerwelt) [05:14:26] (03PS1) 10Dzahn: delete search.wikimedia.org Apache config file [operations/puppet] - 10https://gerrit.wikimedia.org/r/91132 [05:17:15] (03PS1) 10Springle: sideline db1017 for a time, disk & config issues [operations/puppet] - 10https://gerrit.wikimedia.org/r/91133 [05:23:30] (03CR) 10Springle: [C: 032] sideline db1017 for a time, disk & config issues [operations/puppet] - 10https://gerrit.wikimedia.org/r/91133 (owner: 10Springle) [05:30:55] (03CR) 10Physikerwelt: "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/90733 (owner: 10Physikerwelt) [05:35:24] (03PS1) 10Springle: sync sX-master dns after rotations [operations/dns] - 10https://gerrit.wikimedia.org/r/91136 [05:35:55] (03CR) 10Springle: [C: 032] sync sX-master dns after rotations [operations/dns] - 10https://gerrit.wikimedia.org/r/91136 (owner: 10Springle) [07:01:35] (03PS2) 10Ryan Lane: Remove hook definition calls on deployment server [operations/puppet] - 10https://gerrit.wikimedia.org/r/90277 [07:16:28] (03PS1) 10Ori.livneh: EventLogging Ganglia monitor: collect every 10 seconds (was: 30) [operations/puppet] - 10https://gerrit.wikimedia.org/r/91143 [07:20:24] (03CR) 10Ori.livneh: "(6 comments)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/90277 (owner: 10Ryan Lane) [07:22:48] (03CR) 10Ori.livneh: "It's one view *right now*, but that doesn't mean it'll stay that way. I encouraged Juliusz to create this view for start and build it up f" [operations/puppet] - 10https://gerrit.wikimedia.org/r/91079 (owner: 10JGonera) [07:23:14] (03CR) 10Ori.livneh: [C: 032] EventLogging Ganglia monitor: collect every 10 seconds (was: 30) [operations/puppet] - 10https://gerrit.wikimedia.org/r/91143 (owner: 10Ori.livneh) [07:24:32] ori-l: isn't iteritems gone in python 3? [07:24:43] I thought it's deprecated behavior to use iteritems... [07:25:06] for py3, yes. that's a valid reason if you cite it :P [07:25:23] I have to cite every py3 compatibility I use? :) [07:25:42] no, I was half-kidding. It's fine, yeah. [07:25:45] heh [07:26:18] and bools aren't really supposed to be quoted in puppet [07:26:32] if it's indeed a puppet bool, but in that case it should be lowercased [07:26:34] a strict style guide will ding you on it. I should be using a lowercase [07:26:36] indeed [07:27:07] the Python comments were mostly of the 'syntactic sugar you may or may not have known about' variety, but the 'True' thing was confusing [07:27:34] hm, actually a lot of the things I'm doing in get_config could be done using get('item', 'default') [07:27:49] and the last could indeed be setdefault [07:54:46] (03PS3) 10Ryan Lane: Remove hook definition calls on deployment server [operations/puppet] - 10https://gerrit.wikimedia.org/r/90277 [07:55:29] (03CR) 10Ryan Lane: "(6 comments)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/90277 (owner: 10Ryan Lane) [07:56:19] (03CR) 10Ori.livneh: [C: 031] "thanks" [operations/puppet] - 10https://gerrit.wikimedia.org/r/90277 (owner: 10Ryan Lane) [07:56:59] ori-l: so, now adding a repo is down to adding a small amount of config in one hash, and doing a clone on the deployment host [07:57:24] I'll be adding an 'upstream' config item to the hash to automate the clones as well [07:57:37] plus automatically configuring the repos and adding necessary hooks [07:58:43] that's pretty sweet [07:59:12] 'upstream' would be nice, yeah [07:59:45] we have a global config system via pillars, may as well make use of them :) [08:02:05] btw, who wrote puppet-merge? [08:02:18] mediawiki-config needs the same thing [08:02:28] mediawiki-config? [08:02:44] ottomatta mostly wrote it [08:02:55] with help from paravoid, mark, and andrewbogott [08:03:10] yeah, operations/mediawiki-config.git [08:03:26] it's the same process, except not automated [08:03:30] morning [08:03:34] hey hashar [08:03:36] that's slated to be pushed via git-deploy [08:03:56] we expect people to follow the same deploy practice as mediawiki [08:04:03] we have a long involved process for that [08:04:30] but deployment isn't the same as merging [08:05:00] but if there's a workflow for that, so much the better [08:05:08] it's the dev workflow [08:05:20] if you want to change that process have at it, though :) [08:06:51] nah, not really [08:07:22] it's too out of scope for me i think [08:07:28] * Ryan_Lane nods [08:07:54] I try to avoid messing with other's workflow changes when possible ;) [08:12:42] (03PS4) 10Ryan Lane: Remove hook definition calls on deployment server [operations/puppet] - 10https://gerrit.wikimedia.org/r/90277 [08:12:43] (03PS1) 10Ryan Lane: Cleanup of config options and pillars [operations/puppet] - 10https://gerrit.wikimedia.org/r/91151 [08:13:20] (03PS2) 10ArielGlenn: get rid of temp-es* hosts, entries from 2009 long since unused [operations/dns] - 10https://gerrit.wikimedia.org/r/90871 [08:14:08] Ryan_Lane: you should sleep :-] [08:14:13] :) [08:14:27] soon enough [08:14:36] going to do one more simple change [08:15:20] (03CR) 10ArielGlenn: [C: 032] get rid of temp-es* hosts, entries from 2009 long since unused [operations/dns] - 10https://gerrit.wikimedia.org/r/90871 (owner: 10ArielGlenn) [08:16:03] !rt 5676 [08:16:04] http://rt.wikimedia.org/Ticket/Display.html?id=5676 [08:22:57] hashar! you should not be encouraging his sleep >.> [08:23:05] :D [08:23:39] OH MY GOD IT IS CAPSLOCK DAY!! [08:23:41] I FORGOT ABOUT IT [08:23:59] SEE HTTP://CAPSLOCKDAY.COM/ [08:30:40] NO WAY! [08:39:34] MISTAKE, EXCLAMATION MARKS SHOULD BE CONVERTED TO !!!!1111!!BBQ [08:55:53] (03PS1) 10Ryan Lane: Arg lists for fetch and checkout module calls [operations/puppet] - 10https://gerrit.wikimedia.org/r/91155 [08:56:05] and that's my last change of the night :) [09:19:25] WHAT A GREAT DAY!!!1111!!!BBQ [09:28:41] (03PS1) 10Springle: repool db1018 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91156 [09:29:12] (03CR) 10Springle: [C: 032] repool db1018 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91156 (owner: 10Springle) [09:30:15] !log springle synchronized wmf-config/db-eqiad.php 'repool db1018' [09:30:32] Logged the message, Master [09:42:48] !log Zuul: no more voting on code-review label {{bug|55757}} {{gerrit|91157}}. Will make the database happier I guess [09:43:03] Logged the message, Master [09:56:18] PROBLEM - Host sq80 is DOWN: PING CRITICAL - Packet loss = 100% [10:04:13] (03PS1) 10ArielGlenn: removing last vestiges (mgmt ip) for sq31-36, decommed [operations/dns] - 10https://gerrit.wikimedia.org/r/91160 [10:09:19] !log powercycled sq80, hung on mgmt console [10:09:32] Logged the message, Master [10:11:34] RECOVERY - Host sq80 is UP: PING OK - Packet loss = 0%, RTA = 30.99 ms [10:12:22] swapdeath [10:12:29] after 555 days or so not too bad [10:28:45] (03PS1) 10ArielGlenn: fix wrong path for files/charset.cnf (mysql_wmf module) [operations/puppet] - 10https://gerrit.wikimedia.org/r/91162 [10:30:09] (03CR) 10ArielGlenn: [C: 032] fix wrong path for files/charset.cnf (mysql_wmf module) [operations/puppet] - 10https://gerrit.wikimedia.org/r/91162 (owner: 10ArielGlenn) [10:48:50] (03PS1) 10ArielGlenn: update path for icinga/percona files (now in mysql_wmf module) [operations/puppet] - 10https://gerrit.wikimedia.org/r/91165 [10:50:42] (03CR) 10ArielGlenn: [C: 032] update path for icinga/percona files (now in mysql_wmf module) [operations/puppet] - 10https://gerrit.wikimedia.org/r/91165 (owner: 10ArielGlenn) [10:56:23] (03CR) 10Mark Bergsma: [C: 04-1] "Why are you removing the decommissioned Varnish caches from the decommissioned list? They should stay in there for statistics..." [operations/puppet] - 10https://gerrit.wikimedia.org/r/91041 (owner: 10Cmjohnson) [10:58:19] (03PS1) 10ArielGlenn: one more path fixup for icinga/percona files [operations/puppet] - 10https://gerrit.wikimedia.org/r/91167 [10:59:41] (03CR) 10ArielGlenn: [C: 032] one more path fixup for icinga/percona files [operations/puppet] - 10https://gerrit.wikimedia.org/r/91167 (owner: 10ArielGlenn) [11:06:29] PROBLEM - Puppet freshness on tin is CRITICAL: No successful Puppet run in the last 10 hours [11:21:29] PROBLEM - Puppet freshness on mchenry is CRITICAL: No successful Puppet run in the last 10 hours [11:55:56] (03PS1) 10ArielGlenn: fix up path for skrillex.yaml.erb (now in mysql_wmf module) [operations/puppet] - 10https://gerrit.wikimedia.org/r/91173 [11:56:52] (03PS5) 10Andrew Bogott: Remove generic::mysql::packages in favor of mysql module [operations/puppet] - 10https://gerrit.wikimedia.org/r/90194 [11:57:27] (03CR) 10jenkins-bot: [V: 04-1] Remove generic::mysql::packages in favor of mysql module [operations/puppet] - 10https://gerrit.wikimedia.org/r/90194 (owner: 10Andrew Bogott) [11:57:59] (03PS6) 10Andrew Bogott: Remove generic::mysql::packages in favor of mysql module [operations/puppet] - 10https://gerrit.wikimedia.org/r/90194 [11:59:24] (03CR) 10Andrew Bogott: [C: 032] Remove generic::mysql::packages in favor of mysql module [operations/puppet] - 10https://gerrit.wikimedia.org/r/90194 (owner: 10Andrew Bogott) [11:59:43] (03PS2) 10ArielGlenn: fix up path for skrillex.yaml.erb (now in mysql_wmf module) [operations/puppet] - 10https://gerrit.wikimedia.org/r/91173 [12:01:04] (03CR) 10ArielGlenn: [C: 032] fix up path for skrillex.yaml.erb (now in mysql_wmf module) [operations/puppet] - 10https://gerrit.wikimedia.org/r/91173 (owner: 10ArielGlenn) [12:04:49] RECOVERY - Puppet freshness on tin is OK: puppet ran at Tue Oct 22 12:04:41 UTC 2013 [12:13:04] RECOVERY - Puppet freshness on mchenry is OK: puppet ran at Tue Oct 22 12:13:00 UTC 2013 [12:15:53] (03PS1) 10Andrew Bogott: Remove the last of mysql.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/91175 [12:16:28] (03CR) 10jenkins-bot: [V: 04-1] Remove the last of mysql.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/91175 (owner: 10Andrew Bogott) [12:29:55] (03CR) 10Ottomata: "Is there a reason this is called 'log.line.scratch.size' instead of just 'log.scratch.size'? What does log.hash.size refer to? I guess " [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/90028 (owner: 10Edenhill) [12:31:25] (03CR) 10Ottomata: "Note my comment on https://gerrit.wikimedia.org/r/#/c/90028 about the log.* config value names." [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/90030 (owner: 10Edenhill) [12:32:17] (03CR) 10Edenhill: "You are correct in your assumptions and I agree with the namespace clashes." [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/90028 (owner: 10Edenhill) [12:38:25] (03CR) 10Ottomata: "(1 comment)" [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/91052 (owner: 10Edenhill) [12:40:45] (03CR) 10Edenhill: "(1 comment)" [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/91052 (owner: 10Edenhill) [12:55:15] (03CR) 10Ottomata: [C: 031] "This looks great!" [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/91053 (owner: 10Edenhill) [13:01:35] using git fetch --all It hangs on feching gerrit, any idea why? [13:26:40] (03CR) 10Mark Bergsma: [C: 032] Grow scratch pad by temporary buffers if necessary (issue #2) [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/91053 (owner: 10Edenhill) [13:29:09] (03CR) 10Edenhill: "Should probably add "kafka." prefix to kafka properties aswell." [operations/software/varnish/varnishkafka] - 10https://gerrit.wikimedia.org/r/90028 (owner: 10Edenhill) [13:30:51] (03PS2) 10Matanya: Repalce exec calls with file and user. [operations/puppet] - 10https://gerrit.wikimedia.org/r/86889 [13:31:12] (03CR) 10jenkins-bot: [V: 04-1] Repalce exec calls with file and user. [operations/puppet] - 10https://gerrit.wikimedia.org/r/86889 (owner: 10Matanya) [13:34:50] (03PS3) 10Matanya: Repalce exec calls with file and user. [operations/puppet] - 10https://gerrit.wikimedia.org/r/86889 [13:35:11] (03CR) 10jenkins-bot: [V: 04-1] Repalce exec calls with file and user. [operations/puppet] - 10https://gerrit.wikimedia.org/r/86889 (owner: 10Matanya) [13:36:03] (03CR) 10Mark Bergsma: [C: 04-2] "Please don't mix formatting changes (tabs/spaces) with other changes." [operations/puppet] - 10https://gerrit.wikimedia.org/r/86889 (owner: 10Matanya) [13:36:18] (03CR) 10coren: [C: 032] toollabs: Install graphviz [operations/puppet] - 10https://gerrit.wikimedia.org/r/90767 (owner: 10Yuvipanda) [13:37:01] (03CR) 10coren: [C: 032] toollabs: Install ant on -login and -dev [operations/puppet] - 10https://gerrit.wikimedia.org/r/90766 (owner: 10Yuvipanda) [13:37:11] (03PS2) 10coren: toollabs: Sort package names in dev_environ [operations/puppet] - 10https://gerrit.wikimedia.org/r/90765 (owner: 10Yuvipanda) [13:48:45] (03CR) 10coren: [C: 032] toollabs: Sort package names in dev_environ [operations/puppet] - 10https://gerrit.wikimedia.org/r/90765 (owner: 10Yuvipanda) [13:48:56] (03PS2) 10coren: toollabs: Install ant on -login and -dev [operations/puppet] - 10https://gerrit.wikimedia.org/r/90766 (owner: 10Yuvipanda) [13:52:29] (03CR) 10coren: [C: 032] toollabs: Install ant on -login and -dev [operations/puppet] - 10https://gerrit.wikimedia.org/r/90766 (owner: 10Yuvipanda) [13:52:40] (03PS2) 10coren: toollabs: Install graphviz [operations/puppet] - 10https://gerrit.wikimedia.org/r/90767 (owner: 10Yuvipanda) [13:53:40] (03CR) 10coren: [C: 032] toollabs: Install graphviz [operations/puppet] - 10https://gerrit.wikimedia.org/r/90767 (owner: 10Yuvipanda) [14:12:20] (03PS1) 10Hashar: bump upstream to v0.6.0-43-g9d3a3d4 [operations/debs/jenkins-debian-glue] - 10https://gerrit.wikimedia.org/r/91183 [14:12:21] (03PS1) 10Hashar: changelog entry for upstream bump to v0.6.0-43-g9d3a3d4 [operations/debs/jenkins-debian-glue] - 10https://gerrit.wikimedia.org/r/91184 [14:12:24] finally [14:14:25] (03CR) 10Hashar: "The master branch originally created upstream code which we amended to release our own version. I have created an upstream branch that co" [operations/debs/jenkins-debian-glue] - 10https://gerrit.wikimedia.org/r/91183 (owner: 10Hashar) [14:15:14] (03CR) 10Hashar: "debian/changelog entry for the upstream merge made by https://gerrit.wikimedia.org/r/#/c/91183/ ." [operations/debs/jenkins-debian-glue] - 10https://gerrit.wikimedia.org/r/91184 (owner: 10Hashar) [14:30:23] (03PS6) 10Hashar: beta: symlink /a/common [operations/puppet] - 10https://gerrit.wikimedia.org/r/65254 [14:33:23] (03PS4) 10Matanya: Repalce exec calls with file and user. [operations/puppet] - 10https://gerrit.wikimedia.org/r/86889 [14:33:43] (03CR) 10jenkins-bot: [V: 04-1] Repalce exec calls with file and user. [operations/puppet] - 10https://gerrit.wikimedia.org/r/86889 (owner: 10Matanya) [14:34:34] PROBLEM - Host srv291 is DOWN: PING CRITICAL - Packet loss = 100% [14:35:12] (03PS3) 10Hashar: adjust jobrunner/videoscaler role for beta [operations/puppet] - 10https://gerrit.wikimedia.org/r/77034 [14:35:14] i give up [14:35:17] too busy for this [14:38:15] (03CR) 10Hashar: "Job runner configuration should be way easier since it would just be about editing the configuration hash." [operations/puppet] - 10https://gerrit.wikimedia.org/r/77034 (owner: 10Hashar) [14:38:31] (03PS2) 10Hashar: ntp: explicitly reference variable in current scope [operations/puppet] - 10https://gerrit.wikimedia.org/r/86647 [14:41:43] (03PS3) 10Hashar: ganglia wrapper for py plugins (and add diskstat plugin) [operations/puppet] - 10https://gerrit.wikimedia.org/r/85669 [14:57:53] (03PS2) 10Andrew Bogott: Remove the last of mysql.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/91175 [15:00:41] (03PS2) 10Hashar: graphite: make sure we aggregate min/max/count properly [operations/puppet] - 10https://gerrit.wikimedia.org/r/88470 [15:00:47] (03PS2) 10Hashar: graphite: tweak statsd aggregation [operations/puppet] - 10https://gerrit.wikimedia.org/r/88471 [15:25:05] (03CR) 10Andrew Bogott: [C: 032] Remove the last of mysql.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/91175 (owner: 10Andrew Bogott) [15:58:06] csteipp: is https://gerrit.wikimedia.org/r/#/c/90670/ OK to merge? [16:03:55] chrismcmahon: Yeah, just need to merge it on the cluster too when it's merged, so the next deployer doesn't get surprised by it [16:17:39] (03PS2) 10Reedy: Reduce all the www portal docroots into one [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90668 [16:17:48] (03CR) 10Reedy: [C: 032] Reduce all the www portal docroots into one [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90668 (owner: 10Reedy) [16:17:52] (03CR) 10jenkins-bot: [V: 04-1] Reduce all the www portal docroots into one [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90668 (owner: 10Reedy) [16:17:57] SCREW YOU JENKINS [16:18:21] (03CR) 10Reedy: [V: 032] Reduce all the www portal docroots into one [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90668 (owner: 10Reedy) [16:19:21] !log reedy synchronized docroot/wwwportal [16:19:37] Logged the message, Master [16:20:11] (03PS1) 10Chad: Foundationwiki gets new search [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91194 [16:20:43] (03CR) 10Manybubbles: [C: 031] Foundationwiki gets new search [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91194 (owner: 10Chad) [16:21:45] (03CR) 10Chad: [C: 032] Foundationwiki gets new search [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91194 (owner: 10Chad) [16:21:47] (03PS2) 10Reedy: Switch www portals to using one docroot [operations/apache-config] - 10https://gerrit.wikimedia.org/r/90669 [16:21:58] (03Merged) 10jenkins-bot: Foundationwiki gets new search [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91194 (owner: 10Chad) [16:26:09] !log demon synchronized cirrus.dblist 'foundationwiki gets cirrus' [16:26:20] Logged the message, Master [16:27:24] (03CR) 10Dzahn: [C: 031] "dzahn@fenari:~$ apache-fast-test wwwportals.url mw1044" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/90669 (owner: 10Reedy) [16:28:00] (03CR) 10Ottomata: [C: 031] "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/85669 (owner: 10Hashar) [16:28:44] ottomata1: that is a valid point :] [16:30:14] <^d> Reedy: I put foundationwiki in cirrus.dblist, but it doesn't seem to be picking it up. Extension's not installed. [16:30:39] Toch InitialiseSetting and sync? [16:31:29] !log demon synchronized wmf-config/InitialiseSettings.php 'Touch' [16:31:44] Logged the message, Master [16:32:28] <^d> That did it, thanks. [16:34:36] hashar, I can't really tell what's going on for those jenkins-deb-glue changes [16:34:41] but I'm sure its fine, do you need me to merge? [16:35:02] ottomata: basically attempted to update the package [16:35:11] upstream provides a debian directory [16:35:25] so we originally forked their repo in our master branch [16:35:46] I simply added the upstream repo in an upstream branch and merged latest version in our master [16:36:34] ok [16:36:43] (03PS1) 10Reedy: Move www.wikimedia.org portal to wwwportals file [operations/apache-config] - 10https://gerrit.wikimedia.org/r/91195 [16:37:20] ottomata: that is a bit hacky though [16:38:46] so upstream and master are the same? [16:39:48] hashar^? [16:40:12] more or less [16:40:14] (03CR) 10Andrew Bogott: "You can install the pep8 tool locally and check your python files before submitting. Probably there was a .pep8 rule exception allowing t" [operations/puppet] - 10https://gerrit.wikimedia.org/r/90760 (owner: 10Matanya) [16:40:17] master is a fork of some commit of upstream [16:40:25] with 2 commits from Andrew to tweak the package version [16:40:33] then I merged the latest upstream in master [16:40:41] and added another commit in master to further update our version [16:40:41] (03CR) 10Dzahn: [C: 032] Move www.wikimedia.org portal to wwwportals file [operations/apache-config] - 10https://gerrit.wikimedia.org/r/91195 (owner: 10Reedy) [16:41:50] so upstream-branch=upstream (no changes) and debian-branch=master (i know you might not be using git-buildpackage, but whatevs) [16:41:52] I think that sounds good [16:42:19] (03PS3) 10Ottomata: ntp: explicitly reference variable in current scope [operations/puppet] - 10https://gerrit.wikimedia.org/r/86647 (owner: 10Hashar) [16:42:25] (03CR) 10Ottomata: [C: 032 V: 032] ntp: explicitly reference variable in current scope [operations/puppet] - 10https://gerrit.wikimedia.org/r/86647 (owner: 10Hashar) [16:42:40] ottomata: yeah that is it [16:43:04] ottomata: jenkins-debian-glue is being build with …. jenkins-debian-glue which rely on gbp [16:43:17] ottomata: I guess they only use the default so they don't need a debian/gbp.conf [16:43:18] (03PS1) 10Jforrester: cawiki: Enable VisualEditor for Portal: and Viquiprojecte: [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91197 [16:43:19] (03PS1) 10Jforrester: enwiki: Enable VisualEditor for Portal: and Book: [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91198 [16:43:59] ottomata: the package is only used (for now) on a labs instance I play with, so you can put the packages in apt.wm.o without impacts :] [16:44:03] yeah exactly [16:44:05] upstream and master are defaults [16:44:08] ok cool [16:44:17] shall I merge that then? [16:44:25] yup :-] [16:44:41] maybe I will end up setting up my own CI reprepo :-] [16:44:48] (03PS2) 10Jforrester: cawiki: Enable VisualEditor for Portal: and Viquiprojecte: [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91197 [16:45:01] (03PS2) 10Jforrester: enwiki: Enable VisualEditor for Portal: and Book: [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91198 [16:45:05] (03PS1) 10CSteipp: Fix error message for recycled passwords [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/91199 [16:45:39] (03CR) 10Ottomata: [C: 032 V: 032] changelog entry for upstream bump to v0.6.0-43-g9d3a3d4 [operations/debs/jenkins-debian-glue] - 10https://gerrit.wikimedia.org/r/91184 (owner: 10Hashar) [16:45:53] (03CR) 10Ottomata: [C: 032 V: 032] bump upstream to v0.6.0-43-g9d3a3d4 [operations/debs/jenkins-debian-glue] - 10https://gerrit.wikimedia.org/r/91183 (owner: 10Hashar) [16:47:29] (03PS1) 10Andrew Bogott: Remove unused apparmor class [operations/puppet] - 10https://gerrit.wikimedia.org/r/91200 [16:49:25] (03CR) 10Andrew Bogott: [C: 032] Remove unused apparmor class [operations/puppet] - 10https://gerrit.wikimedia.org/r/91200 (owner: 10Andrew Bogott) [16:49:29] (03CR) 10Dzahn: "yep, this is good for Change-Id: Ic236421a6bd178396be2f665ed0948fd10eb48e8 handle them all in one place" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/91195 (owner: 10Reedy) [16:53:30] (03PS3) 10Reedy: Switch www portals to using one docroot [operations/apache-config] - 10https://gerrit.wikimedia.org/r/90669 [16:53:31] (03CR) 10jenkins-bot: [V: 04-1] Switch www portals to using one docroot [operations/apache-config] - 10https://gerrit.wikimedia.org/r/90669 (owner: 10Reedy) [16:54:02] (03PS4) 10Reedy: Switch www portals to using one docroot. [operations/apache-config] - 10https://gerrit.wikimedia.org/r/90669 [16:54:22] ottomata: moving back home… Will upgrade the debian glue package tomorrow if you have made it available in apt.wm.o :-] [16:54:49] cool [16:54:53] oh, you need me to build it? [16:55:17] can you build it and put a .deb on fenari or something? I can put it in apt then [16:59:36] ottomata: yeah will do tomorrow and put on fenari then ping you :-] [17:00:01] ottomata: though git buildpackage might just work :] [17:00:11] I am off of the coworking place, it is closing/ [17:01:05] cool, laters! [17:01:25] (03CR) 10Dzahn: [C: 032] "yep, still works fine after removing all www* docroots" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/90669 (owner: 10Reedy) [17:01:41] wheeeeee [17:03:28] (03PS5) 10Reedy: Move a lot of the miscellaneous wikis out of their own specific docroots [operations/apache-config] - 10https://gerrit.wikimedia.org/r/90703 [17:03:35] (03PS2) 10Reedy: Remove all superfluous docroot folders [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/90704 [17:06:19] !log sync-apache, graceful to unify www portal docroots [17:06:31] Logged the message, Master [17:08:46] :D [17:09:18] Reedy: done [17:09:34] fwif, your topic branch on one of those above is called "(detached" [17:09:57] Awesome [17:10:39] bbiaw [17:10:43] meta wiki and commons is down [17:10:46] well [17:10:51] more of a redirect to foundaiton wiki [17:10:52] RobH: good morning ;) [17:10:58] Oh crap [17:11:03] heyas [17:11:05] I am guessing all wikimedia wikis have suffered this [17:11:10] oh crap [17:11:11] mutante: I think we just broke it [17:11:23] Reedy I caught it instantly :p [17:11:40] I wonder if it was [17:11:40] 205 - [17:11:41] 206 - ServerName wikimedia.org [17:11:41] 207 - Redirect permanent / http://www.wikimedia.org/ [17:11:41] I was typing a detailed post [17:11:42] 208 - [17:11:44] 209 - [17:11:53] ya [17:12:00] arr. revert or can we fix it quicker [17:12:15] right now the caching results are being poisoned [17:12:17] i'd revert. [17:12:18] Probably safer to revert the last commit [17:12:27] images still work I think [17:12:32] when i did this years ago i had to purge caches [17:12:42] Uh, last 2 as the wikimedia one being at fault [17:12:43] https://upload.wikimedia.org/wikipedia/commons/4/43/Wikimood_07.png [17:12:57] Reedy: ping [17:13:04] so can you hurry up? :) [17:13:15] (03PS1) 10Dzahn: Revert "Switch www portals to using one docroot." [operations/apache-config] - 10https://gerrit.wikimedia.org/r/91203 [17:13:35] mark: will varnish need purging for this? [17:13:40] squid [17:13:45] mutante: I'd just reset head a few revisions and push that config [17:13:45] (03CR) 10Dzahn: [C: 032] Revert "Switch www portals to using one docroot." [operations/apache-config] - 10https://gerrit.wikimedia.org/r/91203 (owner: 10Dzahn) [17:14:03] well, atleast theres less of them now.... [17:14:13] less? [17:14:23] mutante, Reedy: I think you have broken Meta-Wiki. [17:14:27] Yes [17:14:28] you may have * [17:14:30] We know full well [17:14:33] Oh, good. :-) [17:14:34] well, some are varnish instead, but i guess capacity is up [17:14:48] syncs [17:14:49] and i suppose im counting uplaod and shouldnt [17:16:39] still redirecting [17:16:47] Yes [17:16:51] No one said it was fixed [17:16:54] ah sorry [17:17:06] I assumed the Dzahn revers fixed it [17:17:15] thats code revert, still has to deploy across cluster [17:17:20] (03PS1) 10Dzahn: Revert "Move www.wikimedia.org portal to wwwportals file" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/91204 [17:17:25] You know what they say about people who assume.... [17:17:28] then we have to manually purge the squid caches since they are poisoned. [17:17:46] (03CR) 10Dzahn: [C: 032] Revert "Move www.wikimedia.org portal to wwwportals file" [operations/apache-config] - 10https://gerrit.wikimedia.org/r/91204 (owner: 10Dzahn) [17:17:55] Marybelle I assume something unpleasant? [17:17:57] :D [17:18:48] RobH how long does that normally take, just to know how much time I have to kill [17:19:16]