[00:00:26] (03PS2) 10Rush: fix sudo stuff for ops/sudoers [operations/puppet] - 10https://gerrit.wikimedia.org/r/134533 [00:00:30] (03PS3) 10Rush: fix sudo stuff for ops/sudoers [operations/puppet] - 10https://gerrit.wikimedia.org/r/134533 [00:00:39] (03PS4) 10Rush: fix sudo stuff for ops/sudoers [operations/puppet] - 10https://gerrit.wikimedia.org/r/134533 [00:02:24] (03CR) 10Springle: [C: 031] puppet3: fix $mariadb dynamic lookup [operations/puppet] - 10https://gerrit.wikimedia.org/r/134374 (owner: 10Giuseppe Lavagetto) [00:02:55] chasemp: eh my notebook went down [00:03:09] fun [00:03:15] but that sudo issue..Nagios user also gets it [00:03:21] see mail to root [00:03:32] yeah should I just merge that then? [00:03:35] fix I mean [00:03:39] or hopeful fix [00:03:40] yea [00:03:52] i am just on phone currently [00:04:05] (03CR) 10Rush: [C: 032 V: 032] "sudo +1'd in irc dzahn" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134533 (owner: 10Rush) [00:04:05] typing on android client and touch screen heh [00:05:52] (03PS1) 10Rush: trying with quotes, have a parse error [operations/puppet] - 10https://gerrit.wikimedia.org/r/134535 [00:06:02] (03PS2) 10Rush: trying with quotes, have a parse error [operations/puppet] - 10https://gerrit.wikimedia.org/r/134535 [00:06:08] (03CR) 10jenkins-bot: [V: 04-1] trying with quotes, have a parse error [operations/puppet] - 10https://gerrit.wikimedia.org/r/134535 (owner: 10Rush) [00:06:10] (03CR) 10Rush: [C: 032 V: 032] "go" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134535 (owner: 10Rush) [00:08:07] chasemp. does the nagios user have to be added? [00:08:07] (03CR) 10Gergő Tisza: [C: 04-1] "If we do things like this, we should update the deployment patches now, because we are bound to forget otherwise." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134343 (owner: 10Gilles) [00:08:39] mutante_: service users are not going to be included for now I think, and it won't bother the existing logic at all [00:09:44] chasemp: just asking because of the mail spam [00:09:53] you saw that yet? [00:10:10] have not [00:10:11] we will get a lot more otherwise [00:10:12] but should be fixed? [00:10:17] looks [00:11:23] (03PS2) 10Gergő Tisza: Introduce finer-grained Media Viewer EventLogging sampling [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134343 (owner: 10Gilles) [00:11:58] chasemp no. still got a new one [00:12:08] once a minute [00:13:38] chase. argon fixed. iridium not? [00:13:39] on argon? [00:13:47] probably just puppet hasn't run [00:13:48] ah [00:13:59] arr.its hard on the tiny screen [00:14:22] yea. just iridium. thats it then [00:14:37] never in my life has a situation demanded a twss so desperately. I will remain totally professional. [00:14:55] ran puppet, seems gtg [00:15:11] thanks [00:22:44] (03PS1) 10Rush: separate users on line for ops [operations/puppet] - 10https://gerrit.wikimedia.org/r/134538 [00:22:59] (03PS2) 10Rush: separate users on line for ops [operations/puppet] - 10https://gerrit.wikimedia.org/r/134538 [00:23:01] (03CR) 10jenkins-bot: [V: 04-1] separate users on line for ops [operations/puppet] - 10https://gerrit.wikimedia.org/r/134538 (owner: 10Rush) [00:23:14] (03CR) 10Rush: [C: 032 V: 032] "should work and be cleaner. no logic change here." [operations/puppet] - 10https://gerrit.wikimedia.org/r/134538 (owner: 10Rush) [00:27:04] (03PS3) 10Krinkle: Introduce finer-grained Media Viewer EventLogging sampling [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134343 (owner: 10Gilles) [00:27:38] (03CR) 10Krinkle: Introduce finer-grained Media Viewer EventLogging sampling (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134343 (owner: 10Gilles) [01:22:55] (03CR) 10BryanDavis: "Fixing things so that the user hacks aren't needed sounds fine. I think that mwdeploy may not be touching NFS in deployment-prep with the " [operations/puppet] - 10https://gerrit.wikimedia.org/r/134519 (owner: 10BryanDavis) [01:24:00] (03PS2) 10BBlack: added Opera support to 514-02 [operations/puppet] - 10https://gerrit.wikimedia.org/r/134490 (owner: 10Yurik) [01:24:08] (03CR) 10BBlack: [C: 032 V: 032] added Opera support to 514-02 [operations/puppet] - 10https://gerrit.wikimedia.org/r/134490 (owner: 10Yurik) [01:26:08] (03PS2) 10BBlack: Remove optional alpha character at the end of the X-CS [operations/puppet] - 10https://gerrit.wikimedia.org/r/134525 (owner: 10Yurik) [01:26:34] (03CR) 10BBlack: [C: 032 V: 032] Remove optional alpha character at the end of the X-CS [operations/puppet] - 10https://gerrit.wikimedia.org/r/134525 (owner: 10Yurik) [01:44:36] (03PS3) 10BryanDavis: [WIP] labs: Fix beta to work with role::mediawiki [operations/puppet] - 10https://gerrit.wikimedia.org/r/134519 [01:50:19] (03CR) 10BryanDavis: "Applied by cherry-pick on deployment-salt. Fixes all puppet runs except videoscaler01.eqiad.wmflabs which has a duplicate definition probl" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134519 (owner: 10BryanDavis) [01:50:35] (03PS4) 10BryanDavis: labs: Fix beta to work with role::mediawiki [operations/puppet] - 10https://gerrit.wikimedia.org/r/134519 [01:53:51] (03PS1) 10Ori.livneh: Don't disable Wikibase when running under HHVM [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134543 [01:54:39] (03CR) 10BryanDavis: [C: 031] "Applied by cherry-pick on deployment-salt. Fixes Aude's problem by creating a /etc/sudoers.d/svn file that mirrors the grants of /etc/sudo" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134491 (https://bugzilla.wikimedia.org/65548) (owner: 10BryanDavis) [01:58:33] (03CR) 10Hoo man: "+1 in general" (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134543 (owner: 10Ori.livneh) [01:59:39] (03CR) 10Ori.livneh: Don't disable Wikibase when running under HHVM (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134543 (owner: 10Ori.livneh) [02:01:08] (03CR) 10Hoo man: [C: 031] "I actually meant that the commit summary doesn't mention Lua Sandbox at all... but whatever." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134543 (owner: 10Ori.livneh) [02:01:22] (03PS1) 10Springle: s4 raise db1056 to normal load, depool db1011 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134544 [02:02:06] (03CR) 10Springle: [C: 032] s4 raise db1056 to normal load, depool db1011 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134544 (owner: 10Springle) [02:02:11] (03PS2) 10Ori.livneh: Don't disable Wikibase or LuaSandbox when running under HHVM [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134543 [02:02:16] (03Merged) 10jenkins-bot: s4 raise db1056 to normal load, depool db1011 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134544 (owner: 10Springle) [02:03:13] my +1 is sticky... magic :P [02:04:26] !log springle synchronized wmf-config/db-eqiad.php 's4 raise db1056 to normal load, depool db1011' [02:04:34] Logged the message, Master [02:06:23] * aude panic [02:06:32] ok, no crisis :) [02:11:49] (03PS1) 10Springle: deploy db1068 to s4 [operations/puppet] - 10https://gerrit.wikimedia.org/r/134545 [02:12:35] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3792 MB (3% inode=99%): [02:13:57] (03CR) 10Springle: [C: 032] deploy db1068 to s4 [operations/puppet] - 10https://gerrit.wikimedia.org/r/134545 (owner: 10Springle) [02:16:39] !log LocalisationUpdate completed (1.24wmf4) at 2014-05-21 02:15:36+00:00 [02:16:44] Logged the message, Master [02:17:11] (03CR) 10Dzahn: "mwalker: pdf2/3 include role::pdf, wikidev group, file mover and your account. role::pdf includes standard,misc::pdf::fonts, ::math, ::pdf" [operations/puppet] - 10https://gerrit.wikimedia.org/r/132136 (owner: 10Dzahn) [02:18:38] !log springle synchronized wmf-config/db-eqiad.php 's4 reduce db1049 load while cloning' [02:18:42] Logged the message, Master [02:19:32] (03CR) 10Dzahn: "..unless it should really be added to the role class.. and we want to check if those packages are installed and touch it in any way.." [operations/puppet] - 10https://gerrit.wikimedia.org/r/132136 (owner: 10Dzahn) [02:19:34] !log xtrabackup clone db1049 to db1068 [02:19:38] Logged the message, Master [02:20:35] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3433 MB (3% inode=99%): [02:24:08] (03PS1) 10Springle: remove db1011 from s4 [operations/puppet] - 10https://gerrit.wikimedia.org/r/134547 [02:25:44] PROBLEM - Puppet freshness on iridium is CRITICAL: Last successful Puppet run was Tue May 20 23:24:25 2014 [02:30:06] !log LocalisationUpdate completed (1.24wmf5) at 2014-05-21 02:29:03+00:00 [02:30:11] Logged the message, Master [02:30:19] (03CR) 10Springle: [C: 032] remove db1011 from s4 [operations/puppet] - 10https://gerrit.wikimedia.org/r/134547 (owner: 10Springle) [02:36:00] springle: those things are related i think (disk space on virt0 and db changes ) [02:36:19] because: root@virt0 /a/backup /a/sqldata [02:36:55] there is some space left, it is just starting to warn [02:37:06] like 3GB [02:38:31] mutante: which db changes? the (r|d)epooling i'm doing? [02:39:02] springle: i thought this < springle> !log xtrabackup clone db1049 to db1068 [02:39:11] but just guessing because of the timing [02:39:19] i thought virt0 complained every time it did a backup, until it removed an old one [02:39:21] i see <+icinga-wm> PROBLEM - Disk space on virt0 [02:39:36] then i see /a/sqldata [02:39:44] i don't think i'm affecting disk space on virt0 in any way. but who knows :) [02:41:00] springle: no no, you're right of course if it has nothing to do with labs(db) [02:42:14] nevermind [02:43:35] PROBLEM - Host ms-be1012 is DOWN: PING CRITICAL - Packet loss = 100% [03:01:07] !log icinga broken on neon due to missing servicegroup 'analytics_eqiad' [03:01:11] Logged the message, Master [03:02:49] (03PS2) 10Dzahn: fix analytics servicegroup name [operations/puppet] - 10https://gerrit.wikimedia.org/r/134375 (owner: 10Filippo Giunchedi) [03:03:17] springle: ^ i'll merge that change by godog [03:03:26] that should fix it [03:04:42] (03CR) 10Dzahn: [C: 032] "merging now because icinga on neon was actually down due to missing 'analytics_eqiad'" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134375 (owner: 10Filippo Giunchedi) [03:04:57] cool [03:05:19] not sure yet why it actually is down [03:05:37] i mean as opposed to failed restart [03:09:00] !log killed db1068 xtrabackup, saturating db1064 network [03:09:05] Logged the message, Master [03:10:20] mutante: i ran puppet on neon and an icinga refresh failed. I tried restarting the service, which failed to come back up, and displayed the nagios group error [03:10:48] should I have not done the second part? [03:13:01] RECOVERY - Disk space on virt0 is OK: DISK OK [03:13:10] !log merging Change-Id: I2827d1ef347 and starting icinga fixed it [03:13:13] springle: it's back [03:13:13] Logged the message, Master [03:13:38] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed May 21 03:12:31 UTC 2014 (duration 12m 30s) [03:13:39] springle: so.. as usual more than one thing came together [03:13:42] Logged the message, Master [03:13:49] there was that existing issue with the wrong service group name [03:13:55] which should have been fixed before [03:14:21] PROBLEM - mysqld processes on db1068 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [03:15:01] but when puppet tries to restart it it uses the "check_config" thing in the init script (reload) [03:15:43] ah, i should have done that instead to see the error [03:15:44] and if that fails it just doesn't do the reload. when you manually restarted it it did not run the test before (restart) [03:16:02] it showed us the error though:) [03:16:18] that got unnoticed before.. and nobody could apply new icinga changes [03:16:42] but to be clear, check_config would have shown me the error too? [03:16:48] and that is fixed now [03:17:14] icinga -v /etc/icinga/icinga.cfg [03:17:37] or "reload" vs "restart" [03:17:38] thanks [03:17:47] reload should also do the check [03:18:18] yes, here [03:18:20] restart => '/etc/init.d/icinga reload', [03:18:29] in the past we had an actual restart there [03:18:42] in a way both have some small disadvantage [03:19:50] something i noticed which seemed odd: [03:20:04] puppet tried the icinga refresh, and failed [03:20:32] later a second puppet run completed successfully, even though by then icinga was down and the previous refresh still hadn't bee done [03:21:11] (at least I think that was the order of events) [03:21:17] i also noticed just now.. that [03:21:29] i ran puppet after merging the fix above [03:21:48] it finished succesfully and also the icinga config was now ok [03:21:58] but puppet failed to actually notice the service needed a "start" [03:22:05] and i manually started it [03:22:12] and then it was all fine [03:22:36] fun [03:22:56] and as a bonus, it _does_ say this [03:23:01] info: Monitor_group[analytics_eqiad]: Scheduling refresh of Service[icinga] [03:23:16] that refresh doesn't appear to start it when it's stopped [03:24:48] we have more to fix . this is unrelated and yet another fun thing [03:24:51] Files::Misc/Exec[fix_icinga_command_file]/returns: change from notrun to 0 failed: /bin/chmod a+rw /var/lib/nagios/rw/nagios.cmd returned 1 instead of one of [0] [03:24:59] but not now anymore.. it works [03:25:41] heh.. i should learn not to touch neon or icinga in the middle of apac shift :) [03:27:25] no worries, it was pre-broken :) [03:28:27] checks for db1072/1073 are in some pending state [03:34:35] springle: it wasn't even done yet, haha, that's how long the puppet run takes there [03:34:53] there = neon? [03:34:57] yes [03:35:00] heh [03:35:16] good old neon [03:35:18] - hostgroups analytics-eqiad [03:35:19] + hostgroups analytics_eqiad [03:35:24] but it's alright and up [03:35:42] Monitor::Service/Service[icinga]: Triggered 'refresh' from 3 events .. even .. so .. [03:36:40] this is the only part left that should hopefully fix itself [03:36:42] https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=all&type=detail&servicestatustypes=1&nostatusheader [03:36:51] the "pending" stuff [03:42:31] !log resume xtrabackup db1049 to db1068, throttled [03:42:34] Logged the message, Master [05:10:50] <_joe_> springle: thanks for the review [05:11:11] :) [05:34:12] PROBLEM - Host db1011 is DOWN: PING CRITICAL - Packet loss = 100% [05:36:02] RECOVERY - Host db1011 is UP: PING OK - Packet loss = 0%, RTA = 0.70 ms [05:38:04] (03PS2) 10Giuseppe Lavagetto: puppet3: fix $mariadb dynamic lookup [operations/puppet] - 10https://gerrit.wikimedia.org/r/134374 [05:41:13] (03PS1) 10Ori.livneh: prevent apache2 service log churn on non-web mediawiki servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/134560 [06:13:30] (03PS2) 10Ori.livneh: prevent apache2 service log churn on non-web mediawiki servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/134560 [06:13:45] (03CR) 10Giuseppe Lavagetto: [C: 032] puppet3: fix $mariadb dynamic lookup [operations/puppet] - 10https://gerrit.wikimedia.org/r/134374 (owner: 10Giuseppe Lavagetto) [06:17:34] <_joe_> springle: merged, let's see if something explodes [06:19:44] (03PS3) 10Ori.livneh: prevent apache2 service log churn on non-web mediawiki servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/134560 [06:20:18] _joe_: puppet run seems fine [06:20:23] (on a relevant box) [06:20:45] <_joe_> springle: it should, I tested that with the compiler [06:20:52] nice [06:25:40] <_joe_> springle: I'm taking a shower, it something bad happens please phone me [06:25:52] ok [06:25:53] <_joe_> you should have my phone number on the contact list [07:03:41] * _joe_ back [07:08:32] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 04:07:45 AM UTC [07:19:02] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Wed May 21 07:18:59 UTC 2014 [07:27:36] (03CR) 10Giuseppe Lavagetto: [C: 031] "Just a comment, then LGTM" (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/134560 (owner: 10Ori.livneh) [07:30:50] (03CR) 10Ori.livneh: prevent apache2 service log churn on non-web mediawiki servers (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/134560 (owner: 10Ori.livneh) [07:39:24] (03PS4) 10Giuseppe Lavagetto: prevent apache2 service log churn on non-web mediawiki servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/134560 (owner: 10Ori.livneh) [07:40:54] (03CR) 10Gilles: "Gergo, I don't think that applying a default to "large" wikis is a good idea. You keep looking at this issue as if a large wiki producing " [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134343 (owner: 10Gilles) [07:43:11] (03CR) 10Gilles: "Thanks for cleaning up the array definitions, Timo." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134343 (owner: 10Gilles) [07:43:15] (03CR) 10Giuseppe Lavagetto: [C: 032] prevent apache2 service log churn on non-web mediawiki servers [operations/puppet] - 10https://gerrit.wikimedia.org/r/134560 (owner: 10Ori.livneh) [07:52:57] (03CR) 10Ori.livneh: [C: 04-1] "needs to be very carefully rebased" [operations/puppet] - 10https://gerrit.wikimedia.org/r/133987 (owner: 10Ori.livneh) [08:01:24] (03CR) 10Ori.livneh: [C: 031] delete mwlib.pp? (pediapress) move to pdf/ocg? [operations/puppet] - 10https://gerrit.wikimedia.org/r/132136 (owner: 10Dzahn) [08:21:00] _joe_: I think I'm on your time zone now! [08:21:56] <_joe_> manybubbles: CEST? [08:45:28] mutante: thanks for merging the icinga change! [08:45:57] (03CR) 10Alexandros Kosiaris: [C: 032] bacula: Also encrypt the data channel [operations/puppet] - 10https://gerrit.wikimedia.org/r/133702 (owner: 10Alexandros Kosiaris) [08:53:37] Guten tag manybubbles [09:01:20] (03CR) 10Filippo Giunchedi: "FWIW, I agree with Ori re: users and the branching, how much work would it be to move those off LDAP?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134519 (owner: 10BryanDavis) [09:03:45] _joe_: yup! for the next week [09:12:12] (03PS1) 10Giuseppe Lavagetto: compare-puppet-catalogs: fix small bugs [operations/software] - 10https://gerrit.wikimedia.org/r/134577 [09:12:17] (03CR) 10jenkins-bot: [V: 04-1] compare-puppet-catalogs: fix small bugs [operations/software] - 10https://gerrit.wikimedia.org/r/134577 (owner: 10Giuseppe Lavagetto) [09:12:19] hi manybubbles just hands up for the unicode change in cirrus, it works well [09:14:24] anyone around - i can't do ssh tools-login.wmflabs.org anymore :-(? [09:14:54] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data exceeded the critical threshold [500.0] [09:15:50] jdlrobson: will check after the 5xx, sec [09:15:56] ok thanks godog [09:16:00] ping me when you are free [09:19:11] <_joe_> godog: are you checking this? [09:19:31] _joe_: yeah, just a brief spike [09:19:43] <_joe_> ok, good [09:20:12] <_joe_> interstingly enough, the anomaly detection did not fire up as the spike was brief and it's configured 'laxed' [09:27:54] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% data above the threshold [250.0] [09:29:28] (03PS2) 10Giuseppe Lavagetto: compare-puppet-catalogs: fix small bugs [operations/software] - 10https://gerrit.wikimedia.org/r/134577 [09:29:31] (03CR) 10jenkins-bot: [V: 04-1] compare-puppet-catalogs: fix small bugs [operations/software] - 10https://gerrit.wikimedia.org/r/134577 (owner: 10Giuseppe Lavagetto) [09:30:49] (03PS3) 10Giuseppe Lavagetto: compare-puppet-catalogs: fix small bugs [operations/software] - 10https://gerrit.wikimedia.org/r/134577 [09:32:02] (03CR) 10Giuseppe Lavagetto: [C: 032] compare-puppet-catalogs: fix small bugs [operations/software] - 10https://gerrit.wikimedia.org/r/134577 (owner: 10Giuseppe Lavagetto) [09:39:28] matanya: thanks! [09:50:19] (03PS1) 10Manybubbles: Change banned Elasticsearch plugins [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134585 [10:01:24] jdlrobson: looks like the instance is responding on ssh but it'll give permission denied as I'm not an admin, what issue are you seeing? [10:02:52] godog: WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! [10:03:09] The RSA host key for tools-login.wmflabs.org has changed, (etc etc) [10:03:23] I need to add the correct host key i guess [10:04:30] jdlrobson: indeed, I see it has changed here https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/tools-login.wmflabs.org [10:36:24] RECOVERY - mysqld processes on db1068 is OK: PROCS OK: 1 process with command name mysqld [10:47:54] PROBLEM - MySQL Processlist on db1064 is CRITICAL: CRIT 86 unauthenticated, 0 locked, 0 copy to table, 0 statistics [10:49:54] RECOVERY - MySQL Processlist on db1064 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 1 statistics [11:07:00] (03PS1) 10Giuseppe Lavagetto: puppet3: fix videoscaler role [operations/puppet] - 10https://gerrit.wikimedia.org/r/134588 [11:08:09] (03CR) 10jenkins-bot: [V: 04-1] puppet3: fix videoscaler role [operations/puppet] - 10https://gerrit.wikimedia.org/r/134588 (owner: 10Giuseppe Lavagetto) [11:18:13] (03PS2) 10Giuseppe Lavagetto: puppet3: fix videoscaler role [operations/puppet] - 10https://gerrit.wikimedia.org/r/134588 [11:19:01] <_joe_> this is another chapter in puppet3 niceties [11:19:13] (03CR) 10jenkins-bot: [V: 04-1] puppet3: fix videoscaler role [operations/puppet] - 10https://gerrit.wikimedia.org/r/134588 (owner: 10Giuseppe Lavagetto) [11:20:32] (03PS3) 10Giuseppe Lavagetto: puppet3: fix videoscaler role [operations/puppet] - 10https://gerrit.wikimedia.org/r/134588 [12:20:08] (03PS1) 10Springle: pool db1068 in s4, warm up [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134594 [12:20:58] (03CR) 10Springle: [C: 032] pool db1068 in s4, warm up [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134594 (owner: 10Springle) [12:21:06] (03Merged) 10jenkins-bot: pool db1068 in s4, warm up [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134594 (owner: 10Springle) [12:22:45] !log springle synchronized wmf-config/db-eqiad.php 'pool db1068 in s4, warm up' [12:22:49] Logged the message, Master [12:25:47] hashar: Hi, do you know if jenkis is still using sqlite... I was looking at the sqlite db connector and some function don't seem to be in the best state [12:26:12] physikerwelt: yes we are using sqlite [12:27:50] that's not a problem ... but I was wondering that mysql uses the myql_ functions and sqllite uses pdo [12:29:17] I have no idea :- [12:29:18] ( [12:30:53] !log Jenkins: updated sysadmin email address from nobody@integration.wikimedia.org to jenkins-bot@wikimedia.org [12:30:58] Logged the message, Master [12:55:39] (03PS1) 10Giuseppe Lavagetto: puppet_compiler: update to 0.2.2 [operations/puppet] - 10https://gerrit.wikimedia.org/r/134596 [12:58:55] (03CR) 10Giuseppe Lavagetto: [C: 032] "a simple version bump." [operations/puppet] - 10https://gerrit.wikimedia.org/r/134596 (owner: 10Giuseppe Lavagetto) [13:07:08] (03PS1) 10Giuseppe Lavagetto: puppet_compiler: assign mime types to produced files [operations/puppet] - 10https://gerrit.wikimedia.org/r/134599 [13:09:54] (03CR) 10Giuseppe Lavagetto: [C: 032] puppet_compiler: assign mime types to produced files [operations/puppet] - 10https://gerrit.wikimedia.org/r/134599 (owner: 10Giuseppe Lavagetto) [13:11:28] damn mailman [13:11:28] it is removing my HTML attachements http://lists.wikimedia.org/pipermail/qa-alerts/2014-May/000004.html :/ [13:14:36] * _joe_ scratchs his head [13:14:42] <_joe_> +e [13:15:34] <_joe_> mmh can't really understand puppet. like, at all. [13:15:50] hashar: link to them instead of mailing html? (I guess the code is there to remove html parts from multipart mails?) [13:16:22] Trminator: something is wrong in the list configuration :-/ But can't figure out what [13:16:35] I disabled content filtering entirely but that does not help hehe [13:19:42] hashar: /q hashar [13:19:44] grml [13:35:32] aude: ping? [13:35:44] hashar: has anything changed with regard to outgoing connections from jenkins. The phpunit tests work fine locally (latexml-integration connects to http://gw125.iu.xsede.org:8888 and receives valid data... but for jenkins this test somehow fails [13:36:28] ? [13:37:03] hi [13:37:14] wikidata question :) [13:37:16] wikidatawiki:sites/SiteList#2014-03-17+Site:2013-01-23 [13:37:19] this is a memcache key [13:37:24] do you happen to know what it is? [13:38:04] it's the contents of the sites table, essentially [13:38:20] what's the issue? [13:38:49] what's the 01-23 there? [13:40:09] self::SERIAL_VERSION_ID . '+Site:' . Site::SERIAL_VERSION_ID; [13:40:27] i suppose if the format of the site object changes, we want to update the key [13:40:44] 03-17 is for the list [13:41:49] self = SiteList [13:42:03] it's 300Mbps of memcached traffic alone [13:42:17] the second one on the same memcache server is 30Mbps [13:42:36] possibly we could do cdb or some other caching [13:42:41] if you recommend something [13:42:45] physikerwelt: are you expected our Jenkins to connect to http://gw125.iu.xsede.org:8888 ? [13:43:07] it's used to add site links to wikidata, to know how items are connected to wikipedias etc. [13:43:28] so used a lot [13:52:54] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data exceeded the critical threshold [500.0] [13:53:02] <_joe_> mh. [13:55:05] <_joe_> a small spike [14:04:06] (03PS1) 10Springle: raise db1068 to normal load [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134602 [14:05:54] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% data above the threshold [250.0] [14:08:49] (03CR) 10Springle: [C: 032] raise db1068 to normal load [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134602 (owner: 10Springle) [14:11:33] !log springle synchronized wmf-config/db-eqiad.php 'raise db1068 to normal load' [14:11:37] Logged the message, Master [14:13:59] (03CR) 10Giuseppe Lavagetto: "Results of the build is noop for the change in puppet 2.7." [operations/puppet] - 10https://gerrit.wikimedia.org/r/134588 (owner: 10Giuseppe Lavagetto) [14:34:19] PROBLEM - Puppet freshness on cp1048 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 02:28:51 PM UTC [14:36:19] PROBLEM - Puppet freshness on cp1048 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 02:28:51 PM UTC [14:38:19] PROBLEM - Puppet freshness on cp1048 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 02:28:51 PM UTC [14:39:07] is someone working on cp1048? [14:39:29] RECOVERY - Puppet freshness on cp1048 is OK: puppet ran at Wed May 21 14:39:23 UTC 2014 [14:41:18] strange, syslog says it was running puppet fine all along, but it cleared when I ran it manually :P [14:41:19] PROBLEM - Puppet freshness on cp1048 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 02:39:23 PM UTC [14:41:59] (03PS1) 10Hashar: contint: rsync host in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/134608 [14:42:12] (for that matter, why did we alert when the last run was only ~10 minutes ago in the alert msg?) [14:43:02] (03CR) 10jenkins-bot: [V: 04-1] contint: rsync host in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/134608 (owner: 10Hashar) [14:43:09] oh, those dates are off by 12 hours, ignore that last comment [14:43:13] hmmmm [14:43:14] (03PS2) 10Hashar: contint: rsync host in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/134608 [14:43:19] PROBLEM - Puppet freshness on cp1048 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 02:39:23 PM UTC [14:43:26] <_joe_> on it [14:43:37] OK: puppet ran at Wed May 21 14:39:23 UTC 2014 / CRITICAL: Last successful Puppet run was Wed 21 May 2014 02:39:23 PM UTC [14:43:40] some kind of tz issue? :) [14:44:00] <_joe_> no, that snmp trap is useless :) [14:44:07] "PM UTC" is awesome too :) [14:44:15] (03CR) 10jenkins-bot: [V: 04-1] contint: rsync host in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/134608 (owner: 10Hashar) [14:44:16] <_joe_> yes [14:44:19] <_joe_> also. [14:45:19] PROBLEM - Puppet freshness on cp1048 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 02:39:23 PM UTC [14:45:25] (03PS3) 10Hashar: contint: rsync host in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/134608 [14:45:29] (03Abandoned) 10Giuseppe Lavagetto: Make nagios hostgroups and servicegroups declarations work. [operations/puppet] - 10https://gerrit.wikimedia.org/r/132395 (owner: 10Giuseppe Lavagetto) [14:47:19] PROBLEM - Puppet freshness on cp1048 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 02:39:23 PM UTC [14:49:19] PROBLEM - Puppet freshness on cp1048 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 02:39:23 PM UTC [14:51:19] PROBLEM - Puppet freshness on cp1048 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 02:39:23 PM UTC [14:51:53] gi11es: ping, SWAT in about 10 minutes [14:52:33] (03PS4) 10Hashar: contint: rsync host in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/134608 [14:53:19] PROBLEM - Puppet freshness on cp1048 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 02:39:23 PM UTC [14:53:49] PROBLEM - Apache HTTP on mw1155 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:09] PROBLEM - LVS HTTP IPv4 on rendering.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:39] RECOVERY - Apache HTTP on mw1155 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.071 second response time [14:54:59] RECOVERY - LVS HTTP IPv4 on rendering.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 66710 bytes in 0.557 second response time [14:55:19] PROBLEM - Puppet freshness on cp1048 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 02:39:23 PM UTC [14:55:21] <_joe_> mh what was that? [14:56:17] http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Image%20scalers%20eqiad&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2&st=1400684158&g=network_report&z=large [14:56:21] our lovely imagescaling infra :) [14:57:13] (03PS5) 10Hashar: contint: rsync host in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/134608 [14:57:19] PROBLEM - Puppet freshness on cp1048 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 02:39:23 PM UTC [14:57:56] (03CR) 10Hashar: "I have deployed it on the labs integration p" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134608 (owner: 10Hashar) [14:59:19] PROBLEM - Puppet freshness on cp1048 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 02:39:23 PM UTC [14:59:22] (03CR) 10Hashar: "I have deployed it on the labs integration puppetmaster. The instance is integration-publisher.eqiad.wmflabs." [operations/puppet] - 10https://gerrit.wikimedia.org/r/134608 (owner: 10Hashar) [14:59:29] RECOVERY - Puppet freshness on cp1048 is OK: puppet ran at Wed May 21 14:59:20 UTC 2014 [15:01:19] PROBLEM - Puppet freshness on cp1048 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 02:59:20 PM UTC [15:01:37] (03CR) 10Andrew Bogott: [C: 032] contint/beta: set natfix for the labs shared proxy [operations/puppet] - 10https://gerrit.wikimedia.org/r/129687 (owner: 10Hashar) [15:01:48] gi11es: Ping for SWAT deploy [15:01:51] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:03:53] (03CR) 10Andrew Bogott: [C: 032] contint: remove ruby-bundler outdated package [operations/puppet] - 10https://gerrit.wikimedia.org/r/126953 (https://bugzilla.wikimedia.org/58040) (owner: 10Hashar) [15:04:33] anomie: pong [15:04:44] (03CR) 10Dzahn: [C: 032] Labs: Add deployment related sudoer rules for svn group [operations/puppet] - 10https://gerrit.wikimedia.org/r/134491 (https://bugzilla.wikimedia.org/65548) (owner: 10BryanDavis) [15:04:51] Ok, starting SWAT [15:05:04] anomie: first time I do one myself, let me know what I need to do [15:05:10] (03CR) 10Andrew Bogott: [C: 032] contint: get composer on Jenkins slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/124305 (owner: 10Hashar) [15:05:54] gi11es: Basically, test that things work properly as I deploy each patch [15:06:00] alright [15:06:12] (03CR) 10Andrew Bogott: [C: 032] contint: install python-requests on all hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/134234 (owner: 10Hashar) [15:06:40] (03CR) 10Dzahn: [C: 032] contint: symlink for Jenkins email templates [operations/puppet] - 10https://gerrit.wikimedia.org/r/129152 (https://bugzilla.wikimedia.org/64232) (owner: 10Hashar) [15:08:01] (03PS1) 10Faidon Liambotis: Add memkeys to role::memcached [operations/puppet] - 10https://gerrit.wikimedia.org/r/134612 [15:08:27] (03PS2) 10Faidon Liambotis: Add memkeys to role::memcached [operations/puppet] - 10https://gerrit.wikimedia.org/r/134612 [15:08:52] (03CR) 10Faidon Liambotis: [C: 032 V: 032] Add memkeys to role::memcached [operations/puppet] - 10https://gerrit.wikimedia.org/r/134612 (owner: 10Faidon Liambotis) [15:09:00] (03CR) 10Andrew Bogott: [C: 031] contint: apply beta natfix on Jenkins slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/127213 (owner: 10Hashar) [15:09:06] (03PS4) 10Hashar: contint: apply beta natfix on Jenkins slaves [operations/puppet] - 10https://gerrit.wikimedia.org/r/127213 [15:09:33] (03CR) 10Hashar: "Rebased manually." [operations/puppet] - 10https://gerrit.wikimedia.org/r/127213 (owner: 10Hashar) [15:10:04] (03PS3) 10Hashar: contint/beta: set natfix for the labs shared proxy [operations/puppet] - 10https://gerrit.wikimedia.org/r/129687 [15:10:17] (03CR) 10Hashar: "Parent change has been rebased as well." [operations/puppet] - 10https://gerrit.wikimedia.org/r/129687 (owner: 10Hashar) [15:12:20] * anomie wonders how long Jenkins is going to take this morning [15:12:38] anomie: Hm.. do I need to list the patches on wikitech, or is creating the cherry-pick enough? [15:12:44] I made one earlier today [15:13:09] Krinkle: List the cherry-pick that needs deploying on wikitech [15:13:27] Just a small one to get statistics on deprecated mw.user methods now instead of in two weeks to aid in migration. [15:13:41] ok https://gerrit.wikimedia.org/r/#/c/134605/ [15:14:05] opsens, the above memkeys utility is cool [15:14:37] e.g. ssh to mc1014 and run "memkeys -i eth0 -d 10.0" [15:15:39] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.004 second response time [15:18:35] !log anomie synchronized php-1.24wmf5/includes/filebackend/FileBackend.php 'SWAT: Replace FOR UPDATE with LockManager use in LocalFile::lock()' [15:18:40] Logged the message, Master [15:19:53] !log anomie synchronized php-1.24wmf5/includes/filerepo/file/LocalFile.php 'SWAT: Replace FOR UPDATE with LockManager use in LocalFile::lock()' [15:19:55] gi11es: Ok, test https://gerrit.wikimedia.org/r/#/c/134505/ please [15:19:56] (03PS5) 10BryanDavis: [WIP] labs: Fix beta to work with role::mediawiki [operations/puppet] - 10https://gerrit.wikimedia.org/r/134519 [15:19:58] Logged the message, Master [15:21:22] anomie: looks fine on commons [15:21:54] gi11es: ok, doing your second change now [15:23:30] (03CR) 10Dzahn: [C: 032] Adjust role::zuul::labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/131438 (owner: 10Hashar) [15:28:36] !log anomie synchronized php-1.24wmf5/includes/filerepo/file/LocalFile.php 'SWAT: Tweaked timestamp logic in recordUpload2' [15:28:39] gi11es: Ok, test https://gerrit.wikimedia.org/r/#/c/134536/ please [15:28:41] Logged the message, Master [15:28:43] Krinkle: You're next [15:29:33] OK [15:29:38] RECOVERY - Puppet freshness on cp1048 is OK: puppet ran at Wed May 21 15:29:28 UTC 2014 [15:29:50] godog: all done with the 'filippo-trusty' VM? [15:31:17] andrewbogott: yep all done, thanks, I can recycle it too [15:31:26] no worries, I'm cleaning house anyway [15:31:47] anomie: looks fine [15:31:54] Krinkle: ok, doing your patch now [15:32:17] gi11es: Thanks [15:32:47] (03Restored) 10Dzahn: retab role/nova.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/127146 (owner: 10Hashar) [15:32:53] hashar: I'm getting this more often recently. Like once or twice a day in random jobs / repositories: https://integration.wikimedia.org/ci/job/mediawiki-core-phpunit-misc/25293/console [15:32:58] git commit missing [15:33:00] race condition? [15:35:03] bug fill it please Krinkle [15:35:14] I am leaving right now for some board meeting [15:35:29] that one is weird :( [15:36:14] Krinkle: sounds like a bug in Zuul. [15:36:21] (03CR) 10Dzahn: [C: 032] retab role/nova.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/127146 (owner: 10Hashar) [15:36:22] Krinkle: maybe the ref refs/zuul/master/Z7c42e240d3234d9398b99f34eefee8dc hasn't been crafted properly [15:40:26] I am out for today [15:40:34] some board meeting this evening [15:42:54] (03CR) 10BryanDavis: "It turns out that the core problem with the user defines is that labs LDAP and Puppet don't agree on some of the details." [operations/puppet] - 10https://gerrit.wikimedia.org/r/134519 (owner: 10BryanDavis) [15:45:02] !log anomie synchronized php-1.24wmf5/resources/src/mediawiki/mediawiki.user.js 'SWAT: Use mw.log.deprecate to track user() and anonymous()' [15:45:04] Krinkle: Test please [15:45:05] Logged the message, Master [15:45:50] anomie: confirmed, calling mw.user.anonymous() from the console triggers a deprecation notice [15:46:06] mediawiki.org and en.wikipedia.org [15:46:56] Krinkle: thanks [15:46:57] (03PS1) 10Rush: admin module deploy cp1037 and es1004 [operations/puppet] - 10https://gerrit.wikimedia.org/r/134625 [15:47:09] !log anomie synchronized php-1.24wmf5/tests/qunit/suites/resources/mediawiki/mediawiki.user.test.js 'May as well sync this too' [15:47:13] Logged the message, Master [15:47:25] * anomie is done with SWAT [15:47:54] (03CR) 10BryanDavis: "I opened bug 65588 to track the l10nupdate gid issue." [operations/puppet] - 10https://gerrit.wikimedia.org/r/134519 (owner: 10BryanDavis) [15:48:36] (03PS2) 10Rush: admin module deploy cp1037 and es1004 [operations/puppet] - 10https://gerrit.wikimedia.org/r/134625 [15:50:20] (03PS3) 10Jforrester: Remove Nearby BF from whitelist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130718 [15:50:30] (03CR) 10Jforrester: [C: 031] Remove Nearby BF from whitelist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130718 (owner: 10Jforrester) [15:56:15] (03CR) 10Rush: [C: 032] "as discussed a few cherry picked hosts from a larger pool, babysitting this and it's just for comprehensive verification, but should have " [operations/puppet] - 10https://gerrit.wikimedia.org/r/134625 (owner: 10Rush) [15:57:34] (03PS6) 10BryanDavis: Labs: Fix beta to work with role::mediawiki [operations/puppet] - 10https://gerrit.wikimedia.org/r/134519 [15:58:07] (03CR) 10BryanDavis: "I opened bug 65591 to track the mwdeploy default shell issue" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134519 (owner: 10BryanDavis) [15:58:14] vim's yaml syntax highlighting is terribly slow... and I mean *terribly*... took over one CPU minute until I killed it for modules/admin/data/data.yaml :/ [15:59:04] hoo: Wow. Same problem here. [15:59:48] is this where I troll you with emacs :D [16:00:25] If I needed an operating system to edit my files... [16:01:15] (03PS2) 10Jforrester: Remove VE formula editor from BF whitelist (graduated) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130719 [16:01:34] (03CR) 10Jforrester: [C: 031] Remove VE formula editor from BF whitelist (graduated) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130719 (owner: 10Jforrester) [16:01:51] hoo: I'm going to try https://github.com/stephpy/vim-yaml to see if it makes things better [16:02:13] bd808: Yeah... read about that as well [16:03:16] hoo: The same file opens instantly with that alternate syntax plugin [16:04:07] bd808: heh... where do I have to stuff that file? Same folder tree in my .vimrc? [16:04:20] * .vim [16:05:01] hoo: I use pathogen so I just needed to clone the git repo into .vim/bundles [16:05:25] if you're adding by hand, put after/syntax/yaml.vim into .vim/after/syntax/yaml.vim [16:05:39] ah, got it [16:10:23] the whole thing looks quite good... better than our current thing, probably [16:11:15] Getting your /home/ stuff wherever you are: [16:11:15] -> if you define a dir for your username in '${module}/files/home' all contents are managed [16:11:15] nice :) [16:11:53] (03PS1) 10Rush: deploy admin for ops to cp* and es* [operations/puppet] - 10https://gerrit.wikimedia.org/r/134631 [16:11:59] .vimrc's for everyone [16:12:09] (03CR) 10jenkins-bot: [V: 04-1] deploy admin for ops to cp* and es* [operations/puppet] - 10https://gerrit.wikimedia.org/r/134631 (owner: 10Rush) [16:12:13] (03PS2) 10Rush: deploy admin for ops to cp* and es* [operations/puppet] - 10https://gerrit.wikimedia.org/r/134631 [16:16:03] (03PS2) 10Giuseppe Lavagetto: retab role/nova.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/127146 (owner: 10Hashar) [16:17:19] (03CR) 10Rush: [C: 031] "+1 for new patchset, unsure if needed but looks good" [operations/puppet] - 10https://gerrit.wikimedia.org/r/127146 (owner: 10Hashar) [16:19:37] (03CR) 10Dzahn: [C: 031] "worked fine on cp1037, i have a home there and can login as me, root login still works as before.. so then it should be fine for the other" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134631 (owner: 10Rush) [16:20:41] (03CR) 10Rush: [C: 032] "off we go!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134631 (owner: 10Rush) [16:21:18] (03Abandoned) 10Jackmcbarn: Restrict the move-categorypages right on enwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/132947 (https://bugzilla.wikimedia.org/65221) (owner: 10Jackmcbarn) [16:23:53] akosiaris, andrewbogott; any word on the trusty image for labs? [16:24:18] mwalker: we're discussing it now, I'll anounce something on labs-l soon [16:24:28] good timing, mwalker! [16:24:47] :) [16:37:40] paravoid, on the topic of operational things relating to PDF; should I be sending performance metrics to statsd, or gmond, or? [16:37:51] statsd would be awesome [16:37:53] mwalker: i replied yesterday [16:38:11] ori, oh... that got lost in my backscroll apparently... /me goes and looks [16:38:32] 18:42 ori: mwalker: to graphite, via statsd [16:38:32] 18:45 ori: mwalker: var buf = new Buffer( 'collection.render:142|ms' ), sock = dgram.createSocket( 'udp4' ); sock.send( buf, 0, buf.length, 8125, 'stats.eqiad.wmnet', sock.close.bind( sock ) ); [16:38:32] 18:46 ori: this may be useful: https://gerrit.wikimedia.org/r/#/c/112647/2/lib/mediawiki.Util.js [16:38:34] 18:46 ori: and this explains what other types of metrics you can send: https://github.com/b/statsd_spec [16:39:15] shiny :D [16:39:18] thanks much! [16:39:41] np [16:41:03] thank you ori :)) [16:41:31] (03CR) 10Ori.livneh: [C: 031] "yes. we should (as a temporary workaround) just call the role mediawiki::role::job_runner (with underscore) or something" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134588 (owner: 10Giuseppe Lavagetto) [16:47:15] <_joe_> ori: do you like that? [16:47:52] like what? [16:48:10] <_joe_> ori: the mind blowing awesomeness of the puppet parser [16:48:25] yeah, i ran into that before [16:50:04] <_joe_> well. I'm off for now, see you later! [16:50:12] bye! [16:50:16] (03CR) 10Alexandros Kosiaris: [C: 032] parsoid: systemuser is only for production [operations/puppet] - 10https://gerrit.wikimedia.org/r/123212 (owner: 10Hashar) [16:50:29] * _joe_ away -one "off duty; I may be around anyway." [16:50:29] (03CR) 10Dzahn: [C: 032] retab role/nova.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/127146 (owner: 10Hashar) [16:50:36] <_joe_> oh snap [17:06:32] ^demon|away: ping me when you arrive? [17:23:18] (03PS1) 10Ori.livneh: move contents of mail.ini to standalone file [operations/puppet] - 10https://gerrit.wikimedia.org/r/134636 [17:34:49] (03PS1) 10Hoo man: Add a warning about running scripts as mwdeploy to misc/maintenance.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/134639 [17:35:01] ^-- comment only change [17:35:04] * hoo eyes ori  [17:35:36] and there's a typo :P [17:36:38] (03PS2) 10Hoo man: Add a warning about running scripts as mwdeploy to misc/maintenance.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/134639 [17:38:28] (03PS1) 10BryanDavis: Fix /a/common/php symlink creation [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134641 (https://bugzilla.wikimedia.org/64748) [17:42:43] (03CR) 10BryanDavis: "The only thing possibly worrying about this is that the ./php symlink will now be absolute rather than relative. I don't think this should" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134641 (https://bugzilla.wikimedia.org/64748) (owner: 10BryanDavis) [17:48:31] (03PS1) 10Ori.livneh: dissolve mediawiki::config::* [operations/puppet] - 10https://gerrit.wikimedia.org/r/134642 [17:52:11] (03CR) 10BryanDavis: Add a warning about running scripts as mwdeploy to misc/maintenance.pp (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/134639 (owner: 10Hoo man) [17:53:10] bd808: We had that topic before mwdeploy can't sudo apache [17:53:32] (03CR) 10Hoo man: Add a warning about running scripts as mwdeploy to misc/maintenance.pp (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/134639 (owner: 10Hoo man) [17:54:16] (03PS2) 10Ori.livneh: dissolve mediawiki::config::* [operations/puppet] - 10https://gerrit.wikimedia.org/r/134642 [17:54:26] hoo: Gah. You're right. I forgot that. [17:54:46] Looks like there a a bunch of cron jobs that should be fixed then. [17:54:50] Yep [17:55:26] I already fixed the wikidata ones... but these things usually require root assistance (chowning files), so I didn't do the other stuff (yet) [17:56:00] They also need to remove the old cron entries I imagine. [17:56:16] any mortal can do that [17:56:22] sudo -u mwdeploy crontab -e [17:56:31] did that before... just be careful :P [17:57:09] hey, can i help or can we fix them in puppet? [17:57:12] that looks quite manual [17:57:22] mutante: needs manual help [17:57:36] I did the wikidata once at the hackathon with andrew otto [17:57:39] to remove old cron jobs? [17:57:39] * ones [17:57:44] what are you fixing [17:57:50] (03PS1) 10Catrope: Fix parsoidCacheUri config in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/134643 [17:57:55] mutante: No, any mortal can remove the old crons [17:57:58] (03CR) 10BryanDavis: Add a warning about running scripts as mwdeploy to misc/maintenance.pp (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/134639 (owner: 10Hoo man) [17:58:02] It's about chowing log files etc. [17:58:45] hoo: are those in maintenance.pp ? [17:58:50] yep [17:58:54] well, partly [17:59:00] set them to ensure => absent to remove them? [17:59:13] that will work with the crons [17:59:25] (03PS3) 10Ori.livneh: dissolve mediawiki::config::* [operations/puppet] - 10https://gerrit.wikimedia.org/r/134642 [17:59:26] but the files still need a manual chown [17:59:31] I wonder if we should add a misc::maintenace::config role that sets Cron { user => apache } and Exec { user => apache } as the default and include it everywhere? [17:59:39] hoo: did they get messed up by human intervention? [17:59:58] the permissions i mean [18:00:01] mutante: Don't think so... but the scripts usually just create them on their own (from mwdeploy) [18:01:03] ok, all i wanted to say is please avoid fixing it manually where possible [18:01:11] it's a cycle [18:02:11] (03PS1) 10Ori.livneh: jobrunners: set nice to 19, not 20 [operations/puppet] - 10https://gerrit.wikimedia.org/r/134644 [18:02:20] You mean cycle of manual intervention needed? [18:03:21] paravoid, yt? [18:03:53] hoo: yea, those quick fixes have the tendency to come back later [18:05:36] Yeah... I'm trying to get stuff straight in puppet to avoid manual messing around :P But if it's about chowing files in accordance to a puppet change... why not per hand [18:06:18] hoo: i may be lacking some background, but why can't we just change the permissions in puppte? [18:06:28] and then run it [18:07:16] mutante: If scripts create their log files themselves (stdout redirect) and then they are owned by mwdeploy:mwdeploy we can just fix them per hand once and then stuff will just be created right [18:08:18] (03PS4) 10Ori.livneh: dissolve mediawiki::config::* [operations/puppet] - 10https://gerrit.wikimedia.org/r/134642 [18:10:08] (03PS1) 10Rush: admin deploy amvslvs1, db72.pmtpa.wmnet, labsdb1001.eqiad.wmnet [operations/puppet] - 10https://gerrit.wikimedia.org/r/134646 [18:11:10] hoo: i see.. so it's about the log files not being created by puppet.. i guess [18:11:20] hoo: understand..yea [18:11:40] Yeah, mostly that [18:11:45] (03CR) 10Rush: [C: 032 V: 032] "self approving this pilot batch as a litmus test of larger changeset" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134646 (owner: 10Rush) [18:11:57] also there's stuff in mwdeploy's home which is kinda awry [18:12:42] hoo: on which host [18:12:57] terbium? [18:12:58] (03Abandoned) 10Hoo man: Use ipv4 when sending log messages to neon [operations/puppet] - 10https://gerrit.wikimedia.org/r/134277 (owner: 10Hoo man) [18:13:01] yes [18:13:21] only two folders, both are logs and very smallish [18:13:35] (03CR) 10Dzahn: "see Change-Id: Iadd9b9786c1a" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134277 (owner: 10Hoo man) [18:13:49] (03PS1) 10Catrope: Add hewiki to Parsoid config in labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/134649 (https://bugzilla.wikimedia.org/65490) [18:14:20] hoo: that's from cron jobs i think [18:14:25] yeah [18:14:27] refreshLinks.. that sounds familiar [18:14:30] i think i even made that [18:14:33] looks [18:14:40] both are defined in that maint.pp [18:14:44] i also remember at one point i suggested [18:14:51] that we move all logs to /var/log/mediawiki [18:14:57] because they are all over the plcae [18:15:04] we have /var/log/wikidata [18:15:05] incl. some in the home [18:15:12] having it grouped makes sense [18:16:09] hold on.. [18:16:14] making a change [18:17:16] (03PS5) 10Ori.livneh: dissolve mediawiki::config::* [operations/puppet] - 10https://gerrit.wikimedia.org/r/134642 [18:17:19] (03PS1) 10Rush: not catching amslvs1 yet for admin [operations/puppet] - 10https://gerrit.wikimedia.org/r/134650 [18:17:42] argg. there used to a pending patch by Reedy as well [18:17:50] from when we talked about that months ago [18:17:58] (/var/log/mediawiki that is) [18:18:05] (03CR) 10Rush: [C: 032 V: 032] "try new match for amslvs1" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134650 (owner: 10Rush) [18:18:14] mh... wasn't CCed on that [18:18:14] (03PS1) 10BBlack: fix daemonization stdio stuff [operations/debs/pybal] - 10https://gerrit.wikimedia.org/r/134651 [18:21:26] (03PS1) 10Dzahn: move refreshLinks cron logs to /var/log/mediawiki [operations/puppet] - 10https://gerrit.wikimedia.org/r/134652 [18:21:33] (03PS6) 10Ori.livneh: dissolve mediawiki::config::* [operations/puppet] - 10https://gerrit.wikimedia.org/r/134642 [18:21:35] (03PS2) 10Ori.livneh: move contents of mail.ini to standalone file [operations/puppet] - 10https://gerrit.wikimedia.org/r/134636 [18:22:11] gerrit died? [18:22:15] ack [18:22:18] was about to say [18:22:20] !log restarting gerrit service [18:22:25] Logged the message, Master [18:22:35] which is surprisingly slow [18:22:49] surprisingly? [18:22:51] ^demon|away: gerrit has fallen over dead man you about? [18:22:51] but still not so slow that I'm not impressed that y'all noticed. [18:22:55] (03CR) 10jenkins-bot: [V: 04-1] move refreshLinks cron logs to /var/log/mediawiki [operations/puppet] - 10https://gerrit.wikimedia.org/r/134652 (owner: 10Dzahn) [18:23:12] heh, ok [18:23:36] (03CR) 10Ori.livneh: [C: 031] "recheck" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134652 (owner: 10Dzahn) [18:23:52] ori: thanks:) stderr: error: The requested URL returned error: 503 [18:23:53] hehe [18:24:30] (03CR) 10Hoo man: [C: 04-1] "If we change this anyway, why not just make it run as apache? No need to put hands on this twice..." [operations/puppet] - 10https://gerrit.wikimedia.org/r/134652 (owner: 10Dzahn) [18:26:52] _joe_: If I want to install the puppet-3 client on labs, is that package available? [18:27:46] andrewbogott: in 14.04 [18:28:10] matanya: it's not that simple, we have a forwardport that is installed by default if I do 'apt-get install puppet' [18:28:12] even on Trusty [18:28:14] hence my question [18:28:31] (03CR) 10Ori.livneh: "verified : http://puppet-compiler.wmflabs.org/change/134642/html/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134642 (owner: 10Ori.livneh) [18:28:42] (03PS1) 10Rush: trying again w/ amslvs1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/134656 [18:28:45] ah, sneaky [18:29:45] andrewbogott: just wondering, if you override mwf mirror and install from ubuntu, would that work [18:30:03] matanya: probably, but doesn't really help me puppetize anything :) [18:30:53] (03CR) 10Dzahn: [C: 031] "that should work, looks pretty identical to the existing "if" right below" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134656 (owner: 10Rush) [18:30:56] (03CR) 10Ori.livneh: "> No need to put hands on this twice..." [operations/puppet] - 10https://gerrit.wikimedia.org/r/134652 (owner: 10Dzahn) [18:30:58] yeah, thanks [18:31:02] (03PS2) 10Ori.livneh: move refreshLinks cron logs to /var/log/mediawiki [operations/puppet] - 10https://gerrit.wikimedia.org/r/134652 (owner: 10Dzahn) [18:31:41] (03PS3) 10Ori.livneh: Don't disable Wikibase or LuaSandbox when running under HHVM [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134543 [18:31:48] (03CR) 10Rush: [C: 032 V: 032] trying again w/ amslvs1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/134656 (owner: 10Rush) [18:31:50] (03CR) 10Ori.livneh: [C: 032] Don't disable Wikibase or LuaSandbox when running under HHVM [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134543 (owner: 10Ori.livneh) [18:32:13] (03Merged) 10jenkins-bot: Don't disable Wikibase or LuaSandbox when running under HHVM [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134543 (owner: 10Ori.livneh) [18:32:57] hm… akosiaris, same question: Is there a package name I can use to override our clever 2.7 hack? I'm trying to install puppet3 on purpose now, and failing. (Both client and master) [18:33:28] (03Restored) 10Dzahn: Move logs to /var/log/mediawiki [operations/puppet] - 10https://gerrit.wikimedia.org/r/83574 (owner: 10Reedy) [18:33:39] ori: hoo: ^ restored Reedy's change :p [18:35:06] but i'm going to do the simple one.. one by one [18:35:39] (finds a lot of abandoned things lately where the only reason was "it took to long") [18:36:10] but then we come back to them anyways [18:36:43] (03CR) 10Dzahn: "restored Change-Id: I2cfbb34d0d1868 that's the one i was referring to" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134652 (owner: 10Dzahn) [18:37:03] In that case it probably makes sense to actually split up the change in to atomic ones [18:37:08] (03PS3) 10Dzahn: move refreshLinks cron logs to /var/log/mediawiki [operations/puppet] - 10https://gerrit.wikimedia.org/r/134652 [18:37:26] especially as the whole manifests/misc/maintenance.pp is probably obsolete [18:38:12] * whole part [18:38:56] (03CR) 10Dzahn: [C: 032] "doing this now, ready to also change the user separately as suggested by Ori" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134652 (owner: 10Dzahn) [18:42:04] (03CR) 10Dzahn: "started actually doing this in Change-Id: If043ed0e33623d that's why i restored" [operations/puppet] - 10https://gerrit.wikimedia.org/r/83574 (owner: 10Reedy) [18:42:47] hoo: ori [18:42:50] root@terbium:/var/log/mediawiki# ls [18:42:50] refreshLinks [18:43:00] there, now we have the directory already [18:43:17] that was fast [18:43:34] you ran it per hand, I guess [18:43:34] puppet? yea [18:43:46] ok... now change it to apache? [18:44:23] Anyone know if there a reason mobile varnish doesn't pass requests with Token cookies, but we do for text varnish? [18:44:44] (03PS1) 10Rush: admin user management amslvs, db*, labsdb hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/134661 [18:44:55] mark / awjr ^: git blames you guys for that code [18:48:10] o_O [18:48:28] (03CR) 10Dzahn: [C: 031] "it worked fine on the other hosts so far.." [operations/puppet] - 10https://gerrit.wikimedia.org/r/134661 (owner: 10Rush) [18:49:44] (03PS2) 10Rush: admin user management amslvs, db*, labsdb hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/134661 [18:49:52] (03CR) 10Rush: [C: 032] "and away we gooooo" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134661 (owner: 10Rush) [18:51:12] csteipp: looks like stuff in puppet's been moved around a bunch since i last looked - what file are you looking at? [18:51:23] (03CR) 10Rush: [V: 032] admin user management amslvs, db*, labsdb hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/134661 (owner: 10Rush) [18:51:52] awjr: templates/varnish/mobile-frontend.inc.vcl.erb [18:52:04] Lines 64-70 [18:52:32] looking [18:53:45] csteipp: don't tokens get set in the sesison? [18:53:50] we pass session cookies [18:54:54] Sortof. If you login with the "remember me" flag, then we set a cookie named somethingSession which expires when the browser closes. [18:55:07] We also set somethingToken, which has a 30 day expiration. [18:55:21] hmm is that new or has somethingToken been there for a while? [18:55:35] for anyone seeing the watchmouse alert re. frdata--it's being fixed [18:55:36] and presumbly that's for the csrf prevention tokens? [18:55:46] The remember me has always worked that way (as long as I"ve been here). Mobile setting that automatically, I"m not sure. [18:56:04] awjr: No, not csrf-- login [18:56:06] yeah - and somethingSession cookies should get passed [18:56:07] ah [18:56:26] huh im not sure csteipp i don't recall any specific conversation around token-specific cookies [18:57:13] awjr: Cool. Worst case is a logged in mobile user would get pages that appear as if they're logged out. [18:57:35] it's possible there's some mobilefrontend magic that checks those cookies client-side but im not sure [18:57:46] awjr: csteipp I am remembering a certain bug, is this relevant ? https://bugzilla.wikimedia.org/show_bug.cgi?id=62614 [18:58:56] csteipp: it's possible we avoided passing that through since we vary the cache on cookies [18:59:19] and that would presumably cause serious fragmentation for anons [18:59:37] (03PS2) 10Dzahn: add tgr to admins::mortals (deployers) [operations/puppet] - 10https://gerrit.wikimedia.org/r/134258 [18:59:43] (03PS3) 10Dzahn: add tgr to stat1003 special accounts [operations/puppet] - 10https://gerrit.wikimedia.org/r/134251 [18:59:54] awjr: Yeah, anons shouldn't ever get a cookie named "Token" (that was the bug GettingStarted hit a while back) [18:59:54] csteipp: mark may remember better than i, although i think i had most of the cookie passthrough conversations with asher [19:00:26] csteipp: also you might check with MaxSem, he may have a better memory about this than i do [19:01:26] well, he did:P [19:01:30] awjr: No problem. I'll probably open a bug and try to check with mark next week on it. [19:01:41] :) [19:03:03] (03CR) 10Dzahn: [C: 032] "tgr: waiting is over" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134258 (owner: 10Dzahn) [19:03:18] (03CR) 10Dzahn: [C: 032] add tgr to stat1003 special accounts [operations/puppet] - 10https://gerrit.wikimedia.org/r/134251 (owner: 10Dzahn) [19:05:44] !log welcome new deployer tgr [19:05:48] Logged the message, Master [19:08:25] \o/ [19:14:52] Krinkle, are you still around? [19:15:17] I'm wondering how parsoid gets deployed to betalabs [19:16:58] PROBLEM - Puppet freshness on cp1058 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 04:16:33 PM UTC [19:19:26] ori, you might now actually; how do git deploy things get deployed onto beta labs? [19:23:02] mwalker: to the best of my knowledge, manually unless otherwise setup [19:23:21] mwalker: deployment-bastion acts like tin in beta [19:23:23] I think gwicke+Krinkle+hashar did something special for parsoid, but it was special [19:23:39] yep; I'm trying to detangle that now :) [19:23:43] Right. Not automated [19:23:45] by manually, I just mean "it's not every 5 minutes like mwcore stuff" [19:23:49] * greg-g nods [19:23:59] (core+extensions) [19:24:07] bd808, that's good to know, I might just have to poke it by hand until hashar and I figure out how to do it manually [19:24:14] parsoid does it's own thing in to deploy in beta [19:24:38] yarpyarp [19:24:55] it's mostly hashar's work [19:25:01] afaik it's deployed on each commit [19:25:13] *nods* it's a post merge trigger [19:25:17] I deploy scap and a couple of logstash related things in beta via manual git-deploy [19:25:47] I'd prefer to use the packages for the beta deploys [19:25:47] I'd like to figure out how to automate git-deploy through jenkins, but haven't poked into it yet [19:27:05] Personally I want to see beta run as much like prod as possible. [19:27:24] our prod deploy system is nothing to write home about [19:27:51] bd808, did you have to do anythign special to the groups? it looks like everything in ocg is owned trebuchet/wikidev -- and I'm not part of the wikidev group so I cannot deploy [19:27:58] PROBLEM - Puppet freshness on virt0 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 04:27:15 PM UTC [19:28:01] No, but beta is our one chance to test things before production [19:28:17] mwalker: Ugh is your primary group svn? [19:28:24] yeeeeuuuup [19:28:25] once beta is more like production in how things are deployed, then you can iterate on beta and then production more easilyr [19:28:28] -r [19:28:45] currently the two are pretty different puppet setups [19:29:12] mwalker: https://bugzilla.wikimedia.org/show_bug.cgi?id=63028 :( [19:29:28] I think the deploy to beta actually involved rsync at some point [19:29:55] gwicke: Beta mostly uses production puppet config these days. Mostly. [19:30:08] yeah, 'mostly' ;) [19:30:16] I don't know about parsoid specifically [19:30:22] there will always be diffs, obviously, but having eg scap do the deploys of cor+ext is obviously good [19:30:42] yeah, that sounds good to me as well [19:30:54] that's the kind of thing I'm talking about :) [19:31:15] in the parsoid case both the beta & prod deploys are broken in different ways [19:31:36] and they are certainly not consistent [19:31:53] mwalker: I can do a git-deploy for you if that helps in the short term. [19:31:54] which is something that should be addressed [19:32:05] they are also very different from our deb [19:32:18] which adds even more ways things can break [19:32:35] what's "they" in that sentence? [19:32:39] both beta and prod? [19:32:41] yes [19:32:57] I'd like to use the same package for everything as much as possible eventually [19:32:58] well... that tells me that the deb is wrong :) [19:33:07] semi-sarcastically [19:33:10] andrewbogott, if you changed my primary group to be wikidev instead of svn; would anything break? [19:33:20] greg-g, prod still uses upstart ;) [19:33:28] which is on its way out [19:33:30] * greg-g nods [19:33:34] mwalker: I don't know. You might not own your files anymore... [19:33:40] If you don't mind being experimented on, we can try it. [19:33:48] andrewbogott, suuure [19:33:55] eh..now it's ok? [19:33:56] sigh [19:34:02] i merged the work around [19:34:25] (03PS1) 10Rush: ams text varnish outliers from cp* admin deploy [operations/puppet] - 10https://gerrit.wikimedia.org/r/134667 [19:34:27] mutante: You merged one work around. There are more bugs because of this. [19:34:43] mwalker: Ok, brace yourself... [19:34:47] mwalker: https://gerrit.wikimedia.org/r/#/c/134491/ [19:35:15] I will gladly submit the revert when https://bugzilla.wikimedia.org/show_bug.cgi?id=63028 is solved [19:35:47] mwalker, try logging out and back in to labs bastion and see how things look? [19:35:58] um… we were talking about labs, right? [19:35:58] PROBLEM - Puppet freshness on cp1045 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 04:35:00 PM UTC [19:36:08] andrewbogott, yes :) I'm trying to git deploy in labs [19:36:38] cp1045: Duplicate definition: Group[wikidev] is already defined in file /etc/puppet/manifests/admins.pp [19:36:43] chasemp: ^? [19:36:46] that related [19:37:15] seems like it would be, let me see what's up [19:38:50] yup I see it, this is the first not-simple (non ops users) case [19:38:54] andrewbogott, fascinating; groups mwalker reports that I'm part of the wikidev group; but I'm still getting permissions errors -- maybe something else is going on in the ocg deploy repo [19:38:58] * mwalker looks more [19:39:23] mwalker: what about files in your homedir -- is it a problem that they're now owned by a group that you're not in? [19:40:28] andrewbogott, hah; yes [19:40:29] (03PS1) 10Rush: dupe wikidev management for some hosts like cp1045 [operations/puppet] - 10https://gerrit.wikimedia.org/r/134669 [19:40:33] mwalker/svn [19:40:52] mwalker: are you able to chgrp them yourself or do you need a root to do that? [19:41:20] I can chgrp them since I'm still the owner [19:41:28] (03PS2) 10Rush: ams text varnish outliers from cp* admin deploy [operations/puppet] - 10https://gerrit.wikimedia.org/r/134667 [19:41:50] (03CR) 10Rush: [C: 032 V: 032] "self merging this cleanup so we can get to 134669 which is needed to correct puppet" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134667 (owner: 10Rush) [19:41:53] mwalker: yes? [19:42:18] (03PS2) 10Rush: dupe wikidev management for some hosts like cp1045 [operations/puppet] - 10https://gerrit.wikimedia.org/r/134669 [19:42:25] (03CR) 10Rush: [C: 032 V: 032] "fix confused puppet on cp1045" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134669 (owner: 10Rush) [19:42:31] mwalker: Mind if I try git-deploy on ocg to see if it works for my account? [19:42:57] Krinkle, I was hoping to duplicate parsoids merge-commit jenkins job to deploy to beta labs, but I'm not familiar enough with the setup to understand how it works so that I can duplicate it [19:42:58] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Wed 21 May 2014 04:41:55 PM UTC [19:43:01] bd808, go for it [19:43:56] mwalker: "Missing the following configuration item: deploy.repo-name" Looks like it didn't bootstrap correctly. I'll see if I can fix it. [19:44:48] andrewbogott: know anything about recent changes to keystone yet ?Must pass glanceconfig to Class[Role::Keystone::Server] at /etc/puppet/manifests/role/keystone [19:44:54] heh; I didn't get that far; I'm still on the `git fetch` step; getting a " cannot open .git/FETCH_HEAD: Permission denied" [19:45:01] which seems easy; but I'm missing what's wrong [19:45:12] andrewbogott: first i thought maybe i caused it, but i only touched "nova" not keystone [19:45:33] Oh, that's on virt1000? I'll look... [19:45:59] it's on virt0 and virt1000 , yea [19:46:06] mutante: so that's only been failing for a few hours? [19:46:10] i had confirmed a puppet run worked on virt1001 [19:46:19] virt1001? [19:46:45] andrewbogott: 18 minutes it seems [19:46:51] mutante: can you link me to your change? [19:47:08] andrewbogott: https://gerrit.wikimedia.org/r/#/c/127146/ [19:47:21] well, hashar's change technically, but merged it earlier [19:47:49] mutante: check out line 321 [19:48:24] andrewbogott: arrg.. that's it..will fix.. sorry [19:49:22] I wonder what the vim command is for 'replace tabs with spaces and also extract a randomly-selected substring while maintaining proper syntax' [19:50:30] andrewbogott: aaaah, that wasn't in PS1, it must have happened on the rebase [19:51:06] it was a complicated rebase _joe_ helped with because that change was so old, it was before the new submodules were created [19:51:11] mwalker: I'm not sure what you're talking about [19:51:16] for tabs vs. spaces, easier to just start from scratch rather than rebase? [19:51:19] * andrewbogott shrugs [19:51:20] mwalker: We already auto-deploy to beta labs in post-merge [19:51:41] yes; for extensions; this is a node service [19:54:32] mwalker: Trebuchet didn't setup the checkout on deployment-bastion properly. I'll try to fix it manually. [19:54:51] (03PS1) 10Rush: convert parsoid users for cp1045 to yaml [operations/puppet] - 10https://gerrit.wikimedia.org/r/134672 [19:57:31] mwalker: Deployed! [20:00:15] bd808, did you manage to `git fetch` or were you just testing `git deploy`? [20:00:16] (03PS1) 10Dzahn: re-add glanceconfig to role::keystone::server [operations/puppet] - 10https://gerrit.wikimedia.org/r/134674 [20:01:00] (03CR) 10Dzahn: [C: 032] re-add glanceconfig to role::keystone::server [operations/puppet] - 10https://gerrit.wikimedia.org/r/134674 (owner: 10Dzahn) [20:01:31] mwalker: `git deploy`. I see there are permissions issues with `git fetch` that need fixing too. [20:02:35] what I cant figure out is why there are perm issues there; I cannot touch anything in the .git folder despite the folder having g+w [20:02:35] for wikidev [20:03:18] mwalker: /srv/deployment/ocg/ocg was missing the g+w bit [20:03:38] RECOVERY - Puppet freshness on virt1000 is OK: puppet ran at Wed May 21 20:03:33 UTC 2014 [20:03:58] RECOVERY - Puppet freshness on virt0 is OK: puppet ran at Wed May 21 20:03:48 UTC 2014 [20:04:29] mwalker: I think perms and trebuchet config are fixed there now. Give it a shot. I lef the repo with origin/master checked out [20:05:43] bd808, hah; it didn't work before I logged out and in again; and now it does [20:05:47] thanks bd808! [20:06:13] I'm writing up a bug about what I had to fix manually. [20:06:53] you might add it to https://bugzilla.wikimedia.org/show_bug.cgi?id=64109 [20:07:05] which is what I had to do manually on tin to get it to work [20:07:51] !log deployed Parsoid 95929801b (deploy sha ae83633a) [20:07:56] Logged the message, Master [20:08:14] mwalker: Weird. That is exactly the bits I had to do on deployment-bastion too. [20:08:14] (03PS1) 10Rush: migrate antimony to admin yaml [operations/puppet] - 10https://gerrit.wikimedia.org/r/134675 [20:08:28] Plus `git submodule update --init --recursive` [20:08:43] aye; that's a known bug that gwicke complains about [20:12:20] mwalker, depending on how long your service takes to restart you might also have to setup a dsh / sudo thing to work around non-root salt timeout issues [20:12:48] in case you are planning to restart your service as part of the deploy ;) [20:13:11] <_joe_> andrewbogott: I was away. In trusty, the puppet package from the distro's archive is puppet 3 [20:13:12] does git deploy service restart not work? [20:13:27] mwalker, only if your service restarts quickly [20:13:55] gwicke, loooovely [20:13:55] _joe_: Is there some way I can ask apt-get to install puppet-3 without monkeying with the sources? [20:13:55] if it doesn't because there's a lot of requests it waits for to finish, only some machines will be restarted [20:13:55] <_joe_> mutante: sorry I must have messed something up in rebasing nova.pp [20:14:25] <_joe_> andrewbogott: no idea, I still haven't looked at our apt architecture [20:14:30] _joe_: no worries, i just wonder how that one single line.. [20:14:57] _joe_: It's not possible to specify a version number with apt? (I always thought it was but am currently failing) [20:15:20] bd808, I think I might need your debugging foo some more -- the branch that it's supposed to be on is 'deploy' so I checked that out; but the sync is failing, presumably because the minions are taking too long to reply [20:15:57] mwalker: Ok. Let me poke at it [20:16:53] mwalker: `git deploy report --detailed sync` says "1/1 minions completed fetch; 0/1 minions completed checkout" [20:17:29] I'll run the checkout command directly on the minion and see what happens [20:17:44] andrewbogott: _joe_ apt-get install puppet= ? [20:17:58] there's the syntax with = [20:18:59] mutante: yeah, it says 'not found' [20:19:15] mwalker: I ran `sudo salt-call deploy.checkout 'ocg/ocg'` from deployment-pdf01 and things look right now [20:19:43] bd808, k; so it was just the minions taking too long... it has to copy down and over a lot of things... [20:19:54] mwalker: It very well may be that the submodule update took longer than the returner was waiting for [20:19:57] * mwalker grumbles about needing *some* better way of doing node deploys [20:19:58] eh, that makes me notice apt.wm and ubuntu.wm are firewalled again? [20:20:01] didn't we fix that [20:20:14] there should be a proxy [20:20:34] http://ubuntu.wikimedia.org/ thinks that should just work [20:20:39] for the general public [20:21:46] mwalker: One thing I'm not so enamored of with Trebuchet is the async timeout issues. That's the root of gwicke's restart problems too. [20:22:43] wow, yea, there is even a ferm rule for http [20:22:51] but no actual iptables rule ? wth [20:24:29] ACCEPT tcp -- anywhere anywhere tcp dpt:http [20:24:34] ehmm.. ok [20:24:53] bd808, the issue is in salt afaik [20:24:59] mwalker: I just did a noop `git deploy` and I had to wait an extra cycle for both the fetch and checkout stages, but it worked. [20:25:18] bd808, https://bugzilla.wikimedia.org/show_bug.cgi?id=61882 [20:25:32] yep; same story on tin now that it has the initial checkout [20:25:55] mwalker: Sounds like the default timeout is too short. [20:26:50] That's just a time that the runner (python script on tin/deploymnet-bastion) waits before deciding that the user should be informed that not all minions have reported back to redis. [20:31:37] (03Abandoned) 10Dzahn: switch torrus over to netmon1001 [operations/dns] - 10https://gerrit.wikimedia.org/r/131915 (owner: 10Dzahn) [20:33:25] (03CR) 10Dzahn: [C: 032] delete mwlib.pp? (pediapress) move to pdf/ocg? [operations/puppet] - 10https://gerrit.wikimedia.org/r/132136 (owner: 10Dzahn) [20:33:58] <^demon|away> chasemp: No, I wasn't about. tldr about gerrit? [20:34:34] ^d: andrewbogott has restared the service it stalled and / or took a minute, not sure why he restarted...chicken or egg? [20:34:47] <^d> Hmmm. [20:35:06] lol @ pdf2 [20:35:09] Suggested packages: nagios2 [20:35:46] ^d, chasemp, I restarted it more or less on a whim -- I made a change similar to https://wikitech.wikimedia.org/wiki/Renaming_users and the gerrit UI was totally ignoring it. [20:35:52] restart caused it to actually reload the db. [20:36:08] So I guess there's ldap, database, and something-else-that-lives-in-memory [20:36:33] andrewbogott: cool deal man, no blame here was just relaying, all seems well now [20:37:01] It was pretty cool how a 30-second outage resulted in multiple queries on IRC :) [20:41:29] ahah! mutante, puppet=3 does not work but puppet=3.* does [20:41:37] well, it doesn't work, technically, but at least it tries [20:42:38] <^d> andrewbogott: There's a way to flush caches. [20:42:42] <^d> That doesn't involve a restart. [20:43:14] (03CR) 10Dzahn: [C: 031] convert parsoid users for cp1045 to yaml [operations/puppet] - 10https://gerrit.wikimedia.org/r/134672 (owner: 10Rush) [20:43:17] ^d: That doesn't surprise me. The googles tell me that there's also a command-line tool for editing users but I guess we don't have it installed. [20:43:32] Anyway, my long-term solution to this plan is redoubling my commitment to never ever renaming anyone [20:43:46] andrewbogott: moar renames! [20:44:07] well, in labs this is. Renaming people in production is my favorite. [20:44:14] andrewbogott: ah:) yea, the full version would be 3.0 or even longer i suppose [20:44:28] 3.4.3-1 [20:44:49] but still doesnt want to install? [20:45:09] <^d> andrewbogott: I think that's 2.9/master and above. [20:45:29] mutante: standard 'but it is not going to be' runaround. I can probably sort it out, just dependency stuff. [20:48:03] (03PS2) 10Rush: convert parsoid users for cp1045 to yaml [operations/puppet] - 10https://gerrit.wikimedia.org/r/134672 [20:48:15] (03CR) 10Rush: [C: 032 V: 032] "go" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134672 (owner: 10Rush) [20:50:39] RECOVERY - Puppet freshness on cp1045 is OK: puppet ran at Wed May 21 20:50:28 UTC 2014 [20:57:51] (03PS2) 10Dzahn: fix "read_only" setting in rsyncd setups [operations/puppet] - 10https://gerrit.wikimedia.org/r/133647 [20:58:56] ^d, are you ready?:) [20:59:18] <^d> Oh yeah that's now. [20:59:29] (03CR) 10MaxSem: [C: 032] GeoData: Switch all wikis to using Elastic as backend [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/133759 (owner: 10Chad) [20:59:40] (03Merged) 10jenkins-bot: GeoData: Switch all wikis to using Elastic as backend [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/133759 (owner: 10Chad) [21:00:11] <^d> da hell is that new bot? [21:00:29] wth? [21:01:01] hehehe [21:01:15] it certainly wants to tell ^d and Mex something [21:01:17] Max [21:01:41] but deploying N/A ..hmm [21:01:52] <^d> Well N/A is the "window" [21:01:58] <^d> Silly bot. [21:02:19] <^d> MaxSem: sync'ing [21:02:45] it parses a wiki page that is also edited by humans? [21:03:01] <^d> nfc. [21:03:21] !log demon synchronized wmf-config/InitialiseSettings.php 'GeoData using Elasticsearch everywhere' [21:03:24] Logged the message, Master [21:04:15] not sure why the event fired five times; but greg-g it works! ish [21:04:43] * ^d adds a new bot to his /ignore list [21:04:45] oh; heh; it actually fired a lot [21:05:04] killing it before it does it again [21:05:04] {"query":{"geosearch":[{"pageid":40678171,"ns":0,"title":"Null Island","lat":0,"lon":0,"dist":0,"primary":""}]}} [21:05:35] mwalker: haha [21:05:58] I apparently need some debounce code... [21:06:29] <^d> MaxSem: #til there's an island called Null. [21:09:51] ^d, seems to work so far [21:10:14] also, found a first problem with data: https://en.wikipedia.org/w/index.php?title=Timeline_of_London&diff=609578893&oldid=606141430 [21:10:17] :P [21:12:32] (03CR) 10Dzahn: [C: 031] "checked keys, UIDs, lgtm" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134675 (owner: 10Rush) [21:16:48] RECOVERY - Puppet freshness on cp1058 is OK: puppet ran at Wed May 21 21:16:38 UTC 2014 [21:19:15] <^d> MaxSem: Well, I think we did it. We'll wait a few days to make sure we don't have to fall back. If things still look good come weekend we'll file some RTs for decom'ing the solr* boxen. [21:19:27] heya MaxSem buddy ole pal. [21:19:48] ^d, do you guys still need these servers? [21:20:16] <^d> Probably. But we'll want to wipe + reprovision them either way. [21:20:28] yeah [21:21:01] MaxSem: when you have 5 minutes lemme know (can't tell if you're in the middle of something, looks like it :) ) [21:21:23] greg-g, actually I'm not anymore:P [21:21:49] cool, so, Max, you know you're one of my favorite deployers right? :) [21:22:07] * greg-g stops with that [21:22:07] * ^d overhears greg-g, finds a corner to pout in [21:22:38] seriously though, I was wondering if you wanted to help with a deploy training session on wed June 4th [21:22:50] ^d: you're just one of my favorites, period. [21:23:05] * greg-g is going to get himself in trouble [21:23:06] <^d> <3 [21:23:39] greg-g, sure [21:23:43] who [21:23:49] 'll be the victim? [21:23:51] * twkozlowski pokes greg-g so that he can poke springle [21:23:55] (03PS1) 10Dzahn: migrate cassandra roots to admin yaml [operations/puppet] - 10https://gerrit.wikimedia.org/r/134733 [21:24:05] twkozlowski: I forget what about? [21:24:11] MaxSem: "everyone" [21:24:14] MaxSem: http://etherpad.wikimedia.org/p/deploytrainingprep [21:24:18] that cswiki collation patch [21:24:33] MaxSem: It'll be in office, but recorded and/or streamed depending on my ability to get google hangouts to do what I want [21:25:07] mmm, deployments in beta... [21:25:10] :) [21:25:23] never actually did that before [21:25:30] just messed with NFS [21:25:37] twkozlowski: hey, start thinking about a special patch you'd like deployed in production during a training session ;) [21:25:52] MaxSem: that's alright, bd808 can do that stuff [21:26:01] MaxSem: I figured you two could tag-team it [21:26:36] twkozlowski: seriously, we were thinking something SWAT-like would be good to show [21:27:11] * twkozlowski runs his 'Actionable shell requests' search [21:27:11] * greg-g wonders who WeAreLegion is [21:27:20] me [21:27:27] twkozlowski: it won't be until June 4th [21:27:30] MaxSem: :) [21:27:47] MaxSem: I can talk about scap until folks are bored to death, but my non-scap deploy experience is literally zero. [21:27:50] wearelegionthedocumentary.com [21:28:12] https://gerrit.wikimedia.org/r/#/c/78016/ greg-g [21:28:25] MaxSem: it'd most likely be at 11am-12:30 or so on June 4th, if that works for you [21:28:30] Waiting since August 7, 2012 [21:28:34] 2013* [21:29:11] so it can wait a couple more weeks :) [21:29:21] twkozlowski, these namespaces are boring [21:29:32] we can invent much funnier ones!:P [21:29:50] Or you can organise the next training on August 7, 2014, so we can make a cake and sing happy birthday to that patch [21:29:51] yeah, it'd be a nice example of a config change, but something involving a backport would be good (which, by definition, we won't be able to come up with until the week of) [21:29:52] (03CR) 10Dzahn: "hey Chase, let's do "admins"/"roots" or "admins"/"root"" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134733 (owner: 10Dzahn) [21:30:18] (03CR) 10Dzahn: "arr, of course "admins/roots" or "admin/root", it already got me now:)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134733 (owner: 10Dzahn) [21:31:19] greg-g: Hmmm... should we do it on Monday instead of Wednesday so we can mess with group0 and not group1? [21:31:55] I guess it doesn't matter as long as we're not inventing things to push [21:32:10] does it really matter? doesn't seem like we need to deploy something serious [21:32:12] (03PS1) 10RobH: testing an hp server [operations/puppet] - 10https://gerrit.wikimedia.org/r/134737 [21:32:30] bd808: worried about your arrival/scatter brain'd-ness on Monday 'tis all [21:32:37] greg-g: Nice of you. [21:32:38] travel-scatter-brain'd-ness, that is [21:33:25] I could drive down and deploy from a starbucks in Nevada. :) [21:33:48] bd808: just no ferrys [21:35:51] (03CR) 10RobH: [C: 032] "small self merged change to add an additional dhcp host entry." [operations/puppet] - 10https://gerrit.wikimedia.org/r/134737 (owner: 10RobH) [21:36:03] greg-g: I'd have to drive up to Seattle to catch a ferry. [21:36:14] bd808: or be real quick and take one in SF bay [21:36:26] or Jersey, I think [21:36:36] Oh yeah. Deploy from Alcatraz! [21:36:40] hah! [21:36:56] that'd be epic [21:37:01] cell service for tmobile is spotty on the bridge [21:37:07] cannot imagine its better from alcatraz [21:37:07] <^d> greg-g: And cold. [21:37:21] do your deployment in a screen session ;] [21:38:12] RobH: mosh+screen ftw [21:38:33] <^d> MaxSem: I've tempted the gods, announced we finished the GeoData migration on the thread to wikitech-l. [21:38:59] whee [21:39:53] * twkozlowski just subscribed qa-alerts to Gmane \o/ [21:40:28] <^d> MaxSem: Unrelated most likely, just happened to notice since I was tailing fatal.log. [21:40:32] <^d> [2014-05-21 21:35:19] Fatal error: Cannot use object of type stdClass as array at /usr/local/apache/common-local/php-1.24wmf4/extensions/MobileFrontend/includes/MobileFrontend.hooks.php on line 299 [21:40:48] <^d> Only saw it like 4 times. [21:41:25] looking [21:44:55] eh, we have like 15 people being jenkins admins , heh [21:45:09] turns out there are 3 groups of them as well [21:45:16] no sudo, some sudo, all sudo [21:45:46] sounds like a hierarchy [21:46:05] yea, trying to translate that to "replacement for old admins.pp" [21:46:08] yaml dta [21:46:10] data [21:46:44] we can do this now https://gerrit.wikimedia.org/r/#/c/134733/ [21:51:43] (03PS1) 10Dzahn: migrate jenkins users/admins/roots to admin.yaml [operations/puppet] - 10https://gerrit.wikimedia.org/r/134739 [21:56:52] (03CR) 10Dzahn: "was it intentional that there are 3 groups of people or an oversight? if they should all have the sudo permissions i'd reduce it to 2 grou" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134739 (owner: 10Dzahn) [22:01:24] (03PS1) 10Dzahn: migrate ytterbium to admin yaml [operations/puppet] - 10https://gerrit.wikimedia.org/r/134741 [22:02:10] (03CR) 10Dzahn: [C: 04-1] migrate ytterbium to admin yaml [operations/puppet] - 10https://gerrit.wikimedia.org/r/134741 (owner: 10Dzahn) [22:18:30] !log Running deleteEqualMessages.php on guwiki (bug 43917) [22:18:35] Logged the message, Master [22:24:10] (03CR) 10Dzahn: [C: 032] "https://wikitech.wikimedia.org/wiki/UID#permission.2Fsecurity_hierarchy" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/134639 (owner: 10Hoo man) [22:24:18] (03PS3) 10Dzahn: Add a warning about running scripts as mwdeploy to misc/maintenance.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/134639 (owner: 10Hoo man) [22:26:36] (03CR) 10Dzahn: [C: 032] Add a warning about running scripts as mwdeploy to misc/maintenance.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/134639 (owner: 10Hoo man) [22:31:00] (03CR) 10Dzahn: "it actually says it should become a daily restart (as opposed to being removed entirely)?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/131333 (owner: 10Matanya) [22:36:25] (03PS1) 10Dzahn: remove 'maerlant' from site.pp/dsh/dhcp [operations/puppet] - 10https://gerrit.wikimedia.org/r/134745 [22:38:07] (03CR) 10Dzahn: "https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=maerlant" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134745 (owner: 10Dzahn) [22:39:31] (03CR) 10Dzahn: [C: 032] ldaplist: Switch to new servicegroups structure [operations/puppet] - 10https://gerrit.wikimedia.org/r/117313 (owner: 10Tim Landscheidt) [22:43:12] (03CR) 10Dzahn: "works as announced on silver in prod" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117313 (owner: 10Tim Landscheidt) [22:43:38] /aways [22:43:42] (03PS2) 10Rush: migrate antimony to admin yaml [operations/puppet] - 10https://gerrit.wikimedia.org/r/134675 [22:44:18] (03CR) 10Rush: [C: 032 V: 032] "had to fix up some merges conflicts locally, no other changes" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134675 (owner: 10Rush) [22:46:00] (03PS1) 10Rush: fix data.yaml from merge conflict [operations/puppet] - 10https://gerrit.wikimedia.org/r/134747 [22:46:29] (03CR) 10Rush: [C: 032 V: 032] fix data.yaml from merge conflict [operations/puppet] - 10https://gerrit.wikimedia.org/r/134747 (owner: 10Rush) [22:46:36] (03CR) 10Rush: "go" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134747 (owner: 10Rush) [22:47:45] (03PS2) 10Rush: migrate ytterbium to admin yaml [operations/puppet] - 10https://gerrit.wikimedia.org/r/134741 (owner: 10Dzahn) [22:47:58] (03CR) 10Rush: [C: 031] "seems good to me" [operations/puppet] - 10https://gerrit.wikimedia.org/r/134741 (owner: 10Dzahn) [22:49:15] (03CR) 10Dzahn: [C: 032] migrate ytterbium to admin yaml [operations/puppet] - 10https://gerrit.wikimedia.org/r/134741 (owner: 10Dzahn) [22:52:27] (03PS1) 10Dzahn: remove wikidev group include, duplicate now [operations/puppet] - 10https://gerrit.wikimedia.org/r/134752 [22:52:55] (03CR) 10Dzahn: [C: 032] remove wikidev group include, duplicate now [operations/puppet] - 10https://gerrit.wikimedia.org/r/134752 (owner: 10Dzahn) [22:55:38] why you all flooding :) [23:02:19] greg-g, who is suiting up? [23:02:36] mwalker: no bot pings! [23:03:22] MaxSem: ori got your swat attire? [23:03:29] anything to deploy? [23:03:35] spagewmf has something [23:04:02] and tgr apparently [23:04:16] greg-g, I decided to test it in a sandbox in case it went off the rails again [23:04:19] oh, yeah, /me refreshes again [23:04:26] mwalker: how'd it go? [23:04:38] greg-g: added at 15:59 [23:04:41] totally legit [23:04:55] greg-g, it didn't blow up :D [23:05:19] tgr: :P [23:07:55] BTW I updated https://wikitech.wikimedia.org/wiki/SWAT_deploys with `For extension fixes, the SWAT team prefers that the requestor submits a gerrit change to core that "bumps" the extension submodule to incorporate the fix.' [23:08:10] thanks spagewmf [23:11:21] MaxSem: you seem busy other places? [23:11:46] I can do it [23:11:59] kk [23:13:04] * MaxSem bites tgr [23:16:16] ^d: lots of cirrusSearchLinksUpdateSecondary jobs on wikidatawiki, is that expected? [23:16:21] !log maxsem synchronized php-1.24wmf5/extensions/Flow 'https://gerrit.wikimedia.org/r/#/c/134746/' [23:16:26] Logged the message, Master [23:16:40] spagewmf, ^^^ [23:17:03] it is found that MaxSem is to be thanked. [23:17:11] :) [23:17:19] MaxSem: in my defense, that guideline was added 20 minutes before the start of the SWAT [23:17:33] probably not enough time to learn git submodules :) [23:18:44] tgr you're not alone, most of the SWAT gerrits I checked were just the extension changes, hence my query to the team [23:22:27] !log maxsem synchronized php-1.24wmf5/extensions/MultimediaViewer/ 'https://gerrit.wikimedia.org/r/#/c/134750/' [23:22:32] Logged the message, Master [23:22:48] tgr, ^^^^ [23:24:40] MaxSem: doesn't seem to have any effect... I suppose ResourceLoader can introduce some delay? [23:24:47] yep [23:25:05] also, most wikis are on wmf4 [23:25:19] I'm looking at mediawiki.org [23:26:46] tgr, try appending ?debug=1 and/or Shift-reload [23:28:04] it arrived [23:28:10] thanks, MaxSem! [23:28:14] !log maxsem synchronized php-1.24wmf5/extensions/MultimediaViewer 'touch' [23:28:16] :) [23:28:19] Logged the message, Master