[02:27:45] !log LocalisationUpdate completed (1.21wmf9) at Mon Feb 11 02:27:44 UTC 2013 [02:27:48] Logged the message, Master [02:52:40] !log LocalisationUpdate completed (1.21wmf8) at Mon Feb 11 02:52:39 UTC 2013 [02:52:42] Logged the message, Master [10:37:01] New patchset: Hashar; "(bug 44424) wikiversions.cdb for labs" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47564 [11:50:27] New patchset: Silke Meyer; "Fixing a typo in mx-extension" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48439 [11:56:04] New review: Nikerabbit; "mx -> mw?" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/48439 [11:58:14] New review: Silke Meyer; "Erm... yeah. Fixing a typo and adding one. m)" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/48439 [11:58:22] New patchset: Silke Meyer; "Fixing a typo in mw-extension" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48439 [12:03:09] New patchset: Silke Meyer; "Fixing permissions on images folder" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48441 [12:20:48] New review: Silke Meyer; "I was also wondering if this shouldn't be in mediawiki.pp as it is related to permissions on the def..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/48441 [12:53:57] New patchset: Legoktm; "(bug 44871) Enable Translate extension for foundationwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48444 [13:03:53] New review: Siebrand; "Please don't merge this before there is ample justification for this on the bug." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/48444 [13:55:22] !log jenkins: regenerating all phpcs-HEAD jobs so they stop failing when no files need to be processed {{bug|44567}} {{gerrit|48450}} [13:55:26] Logged the message, Master [14:56:14] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47074 [14:59:09] !log hashar synchronized wmf-config/CommonSettings.php '{{bug|44251}} hardcode $wgDBuser = "wikiuser" {{gerrit|47074}}' [14:59:11] Logged the message, Master [14:59:45] mw1161: rsync: change_dir#3 "/apache/common-local" failed: No such file or directory (2) [14:59:45] mw1161: rsync error: errors selecting input/output files, dirs (code 3) at main.c(643) [Receiver=3.0.9] [14:59:46] bahhh [14:59:49] poor mw1161 [15:00:26] New review: Hashar; "deployed" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47074 [15:15:27] backportpackage: Error: The package 'python-tox' does not exist in the Ubuntu primary archive in raring, raring-security, raring-updates or raring-proposed [15:15:33] but it is there http://packages.ubuntu.com/raring/python-tox !!! :D [15:33:26] dpkg-buildpackage: full upload (original source is included) [15:33:27] yeahhh [15:33:33] echo $? [15:33:33] 0 [15:33:35] ahaha [15:33:38] * hashar chubabba dance [15:36:29] hehe [15:36:40] I end up with a bunch of files [15:36:48] which are supposed to build a package for me now :-] [15:36:49] ping paravoid for this yummy bit of fun? [15:36:50] https://gerrit.wikimedia.org/r/#/c/48041/ [15:36:51] :) [15:36:54] https://bugzilla.wikimedia.org/show_bug.cgi?id=44443#c1 [15:36:58] that is a lot of funny fun [15:44:34] hmm [15:44:42] I think I will end up creating a jenkins job to backport packages :-D [15:44:58] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Package[build-essential] is already defined in file /etc/puppet/manifests/generic-definitions.pp at line 1073; cannot redefine at /etc/puppet/manifests/misc/package-builder.pp:11 on node integration-jobbuilder.pmtpa.wmflabs [15:44:59] grmbmbmb [15:45:32] !g I6a4e9ef4c3a70d9484ae60362b81bb35992cb477 [15:45:32] https://gerrit.wikimedia.org/r/#q,I6a4e9ef4c3a70d9484ae60362b81bb35992cb477,n,z [15:48:23] New patchset: Ottomata; "Fixing duplicate package declaration on stat1" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48458 [15:48:30] haha [15:48:37] i'm fixing a similiar error but for a different package [15:49:32] New patchset: Ottomata; "Fixing duplicate package declaration on stat1" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48458 [15:50:04] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48458 [15:50:11] ottomata: my instance was outdated :-] [15:50:26] puppet should probably detect that they are the same [15:50:27] p [15:50:40] I mean when you have two package { 'foobar': ensure => present } [15:56:49] oh [15:56:52] so [15:57:03] apparently we can detach a process from a terminal [15:57:22] ^Z [15:57:22] bg [15:57:26] disown 1 [15:58:45] * hashar waves [15:58:47] see you later [15:58:48] huh [15:58:51] did not know that [15:58:52] ok byyee [16:02:45] New patchset: Ottomata; "Adding Frank Schulenburg user and adding him on stat1. RT 4475" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48460 [16:03:19] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48460 [16:05:09] New patchset: Aude; "Enable Wikibase Client on enwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48461 [16:05:52] ottomata the RT man should name an heir :) [16:06:15] yeah who's on it htis week? [16:06:21] I dunno! [16:06:36] New review: Aude; "i'm doing this out of order.... have a couple backports to do for the extension." [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/48461 [16:21:39] New patchset: Cmjohnson; "Decommissioning srv226-234 updating site.pp/decommissioning.pp and dhcpd file" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48464 [16:22:56] Change merged: Cmjohnson; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48464 [16:27:40] !log commenting out srv226-234 on pybal/pmtpa [16:27:41] Logged the message, Master [16:27:49] New patchset: Jgreen; "adding r-cran-rmysql to stat1 for RT #4470" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48467 [16:28:35] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48467 [16:36:22] New patchset: Demon; "Adding torblock to debug log groups" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48469 [16:39:06] Change merged: Demon; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48469 [16:40:18] New review: Aude; "https://gerrit.wikimedia.org/r/#/c/48465/ has our backports. once that is merged, then we are okay ..." [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/48461 [16:40:40] !log demon synchronized php-1.21wmf9/extensions/TorBlock 'Updating TorBlock to master' [16:40:42] Logged the message, Master [16:41:00] * aude eagerly awaits enwiki deployment :) [16:41:28] !log demon synchronized wmf-config/InitialiseSettings.php 'Enable logging group for torblock' [16:41:29] Logged the message, Master [16:41:46] <^demon> apergos: Ok, so TorBlock & InitialiseSettings changes are live. [16:41:56] sweet [16:42:01] now we just wait for it to fail [16:42:19] thank you! [16:42:48] <^demon> Bah, way too much logging. [16:43:07] <^demon> ssh to fluorine, and tail /a/mw-log/torblock.log [16:46:09] ok I won't merge that then :-D [16:46:14] sec [16:46:53] <^demon> I'm disabling some of the useless logging. [16:46:55] well that is really a lot og logging [16:46:58] yeah, saw that [16:47:22] I mean [16:47:27] "Checking Tor status"... really? [16:48:49] <^demon> Yeah, I'm removing those. [16:49:05] <^demon> "Unable to load Tor exit node list: cold load disabled on page-views." [16:49:07] <^demon> We don't care. [16:49:20] yep [16:50:31] !log demon synchronized php-1.21wmf9/extensions/TorBlock 'TorBlock to master, now with less annoying logs' [16:50:32] Logged the message, Master [16:51:04] heh [16:51:08] <^demon> apergos: Could you clear that log so we can start clean? [16:51:12] New patchset: Silke Meyer; "Fixing broken dependencies for updating mw-extensions" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48474 [16:51:41] done [16:52:01] <^demon> Ok, now let's see what happens when we try running without --force. [16:52:13] well it's working atm [16:52:19] so hopefully... nothing [16:52:25] <^demon> Figures. Well, now we've got a way to debug next time :) [16:52:38] yep [16:52:56] thanks for setting this up! [16:54:44] ^demon: on wikidata, we're having problems with the new minwiki (new wikipedia) appearing in the list of sites [16:55:02] <^demon> apergos: np. [16:55:02] as seen in JS, it seems, but memcached and api are okay [16:55:16] any idea how to fix? to touch a file? [16:55:23] <^demon> Dunno, let's find out. [16:55:31] ok [16:55:44] <^demon> apergos: Gerrit upgrade in 8 hours :D [16:55:44] it's WikibaseLib/includes/modules/SitesModule.php [16:55:54] i suspect that's the one that needs to be "kicked" [16:56:05] it's a dynamically generated module base on the sites table [16:56:13] <^demon> So, touch & sync? [16:56:19] sure [16:56:38] the memcached was fine, on friday and it appears in the api [16:57:00] woo hoo! [16:57:12] drinks in 8.5 [16:58:23] hmmm, i don't see minwiki in http://wikidata.org/w/api.php [16:58:38] !log demon synchronized php-1.21wmf9/extensions/Wikibase/lib/includes/modules/SitesModule.php 'Touching file' [16:58:39] Logged the message, Master [16:59:29] * aude sees if that fixes it [16:59:41] New patchset: CSteipp; "Enable Global Abuse Filters on test, mediawiki, wikidata" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48070 [17:02:44] still no min wikipedia :( [17:02:59] <^demon> I see 'min' under several language code listings. [17:03:04] <^demon> Ctrl+F for "min," [17:03:30] <^demon> Oh hrm, but not under sites. [17:03:32] <^demon> Got it. [17:03:42] in the wikibase api modules? [17:04:31] <^demon> Yeah, I'm looking at wbgetentities [17:04:37] wbsetsitelink [17:04:39] hmmm [17:04:55] <^demon> minwiki is new-ish, right? [17:05:12] it's there as a language [17:05:15] <^demon> Were the interwiki caches rebuilt? [17:05:33] i doubt it or no idea really [17:06:07] <^demon> I doubt they were. Hmm, how to do this. [17:06:08] shouldn't matter for the wikibase modules though [17:06:39] <^demon> How do we populate the sites table? [17:07:02] we already did and it's there [17:07:11] The interwiki caches were updated a couple of times post addition of the new wikis [17:07:12] e.g. when i look on the toolserver and reedy checked [17:07:22] and the memcached seems refreshed, or maybe not [17:07:40] <^demon> Hmm. [17:08:06] it would be cool if rebuilding interwikis also repopulated the sites table [17:08:56] Probably just stick a new hook in the maintenance script.. [17:09:16] sure [17:09:24] sure it's not difficult to do [17:11:17] http://www.wikidata.org/wiki/Special:ItemByTitle/minwiki/Laman_Utamo [17:11:32] there it should offer "You can also create an item." [17:11:50] but does not, which indicates it didn't like the site code [17:12:06] * aude pokes at the code and see how to debug [17:14:53] New review: Legoktm; "LGTM, but your commit message still mentions wikidata though." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/48070 [17:18:45] Change abandoned: Demon; "Forget this. There's talk of scrapping wikibugs entirely (since wm-bot also does BZ support)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/45322 [17:34:42] New patchset: CSteipp; "Enable Global Abuse Filters on test, mediawiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48070 [17:38:08] ^demon: Reedy for debugging stuff with debug log, is it preferred to use wfDebug or wfDebugLog? [17:38:34] <^demon> wfDebugLog() is nice, then we can send things to their own logs with $wgDebugLogGroups [17:38:37] * aude can stick a debug point into the code [17:38:39] ok [17:49:27] good morning Ryan_Lane! you are probably checking monday emails, but have some probably easy LDAP questions for you, whenever you get a sec [17:50:28] ottomata: ? [17:50:43] New patchset: MaxSem; ".gitreview" [operations/debs/mod_tile] (master) - https://gerrit.wikimedia.org/r/48478 [17:52:19] i'm looking into hadoop ldap group lookups, so we can disable nss on the analytics machines [17:52:28] ok [17:52:33] they are documented here [17:52:34] http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml?view=markup&pathrev=1302740 [17:52:39] the relevant configs [17:52:49] i'm trying to construct an ldap search with the proper values first [17:52:53] so I can figure out what to set them to [17:52:57] I think I know most of the details [17:53:06] i'm mainly unsure on this one [17:53:10] hadoop.security.group.mapping.ldap.search.filter.group [17:53:22] hadoop.security.group.mapping.ldap.search.filter.group [17:53:22] 171 [17:53:22] (objectClass=group) [17:53:22] 172 [17:53:22] [17:53:22] 173 [17:53:23] An additional filter to use when searching for LDAP groups. This should be [17:53:23] 174 [17:53:24] changed when resolving groups against a non-Active Directory installation. [17:53:24] 175 [17:53:25] posixGroups are currently not a supported group class. [17:53:30] question1 [17:54:03] New patchset: MaxSem; ".gitreview" [operations/debs/osm2pgsql] (master) - https://gerrit.wikimedia.org/r/48479 [17:54:08] should I try to lookup the user group membership in the ou=project or ou=posixgroups (i think ou=project, since I can do direct LDAP lookup and don't have to worry about the unix group sync) [17:54:10] ? [17:54:27] hold on, I need to get coffee before I read any of this [17:54:54] haha, ok that's what I thought :p [17:54:58] ping me whenever you are ready, no hurry [17:56:34] New patchset: MaxSem; ".gitreview" [operations/debs/osm-mapnik-style] (master) - https://gerrit.wikimedia.org/r/48480 [17:57:13] ottomata's going for another week? :) [17:58:05] eh? [17:58:17] <^demon> Ryan_Lane: T minus 7 hours :D [17:59:33] ^demon: is a package already built? [17:59:40] ottomata: RT [17:59:58] <^demon> Ryan_Lane: Not yet, but all the changes for it are in gerrit. [18:00:13] <^demon> https://gerrit.wikimedia.org/r/#/c/47806/ [18:00:57] ^demon: ok. need me to build the package, or can you? [18:01:18] <^demon> If you would, that'd be great. I'm a packaging noob. [18:01:23] posixgroups are currently not a supported class? :D [18:01:45] ottomata: that's not an issue really [18:01:49] can't wait until i can edit my commit summaries :) [18:01:55] hm [18:01:56] maybe it is [18:02:23] oh. no. it's fine [18:02:37] ottomata: we use both posixgroup and groupofnames objectclasses [18:02:51] <^demon> As soon as the package is good and on brewster, I can do the apt-get upgrade on labs (the one part of the update I haven't tested yet) [18:03:01] yeah, Ryan_Lane, i'm not sure which, i thikn groupofnames would be better [18:03:05] it should be: (objectClass=groupofnames) [18:03:13] well, it apparently has to be ;) [18:03:30] posixgroup, depending on the rfc doesn't provide usernames as full dns [18:03:38] hmm, ok [18:03:41] we're using an rfc that does, so it should also be fine [18:03:42] if I try ldapsearch [18:03:44] on the CLI [18:03:49] '(objectClass=groupofnames)' cn=analytics [18:03:50] is that ok? [18:03:55] eh? [18:03:57] ok [18:04:04] why cn=analytics? [18:04:07] is that an ldap group? [18:04:09] <^demon> Ryan_Lane: I'll be back soon, gonna grab some lunch. [18:04:15] project yeah [18:04:23] ah. project-analytics [18:04:28] ottomata: so..... [18:04:29] with just '(objectClass=groupofnames)' [18:04:34] it returns tons all of them [18:04:35] be careful there [18:04:47] that group may have people in it that haven't signed an NDA [18:04:51] right [18:04:53] so [18:04:58] i just CCed you on an RT [18:05:03] that has been sitting for a while [18:05:06] the analytics one is just my test [18:05:15] the has-signed-nda group? [18:05:22] lemme find link [18:05:33] https://rt.wikimedia.org/Ticket/Display.html?id=3770 [18:05:34] yup [18:06:15] but, aside from that, '(objectClass=groupofnames)' makes sense. and hadoop shoudl know however the ldap query is supposed to be constructed. [18:06:21] i was just trying to do it via the cli so I could see it myself first [18:06:51] but, also, since there is no other cn=analytics match [18:06:57] when I do ldapsearch with only that as the filter [18:07:00] i get the expected result [18:07:08] without objectClass=groupofnames [18:07:28] how do you see the has-signed-nda group working? [18:07:46] it's cn=project-analytics [18:12:00] hmm [18:12:06] but project-analytics is the posixgroup, right? [18:12:22] re. has-signed-nda (i'd prefer to call it nda-data, nda-) [18:12:38] anyone who has signed the Data NDA would be added to the nda-data project [18:13:05] sensitive files in hdfs would be group-owned by nda-data (with proper mode) [18:13:17] it has two objectclasses [18:13:26] ah ok [18:13:29] hadoop looks up user group membership in ldap, etc. [18:13:33] * Ryan_Lane nods [18:13:49] so, cn=project-analytics has both posixgroup and groupofnames objectclasses [18:13:58] cn=project-analytics does too?, hmmmm [18:14:00] hmm [18:14:01] yes [18:14:20] cn=analytics under the project ou does no have posixgroup [18:14:25] *not [18:14:26] dn: cn=project-analytics,ou=groups,dc=wikimedia,dc=org [18:14:26] objectClass: groupofnames [18:14:26] objectClass: posixgroup [18:14:33] vs [18:14:46] # analytics, projects, wikimedia.org [18:14:46] dn: cn=analytics,ou=projects,dc=wikimedia,dc=org [18:14:54] don't use ou=projects [18:14:59] it's only used for openstack now [18:15:14] hmmmm [18:15:15] ok [18:15:44] ok, so the project-analytics one is the one that we set up to manually sync from the ou=projects thing, right? [18:15:58] so if we use that, I should group own the hdfs files by project- [18:16:00] project-nda-data [18:16:02] right? [18:17:42] I think it'll be a regular ldap group [18:17:46] that doesn't start with project- [18:19:37] hm [18:20:16] but the only cn=analytics entry I see is the one with ou=projects [18:20:58] right? just so I'm sure I'm remember this correctly [18:21:23] ou=project: analytics [18:21:23] ou=groups project-analytics [18:22:19] we sync from the ou=project cn=analytis to ou=groups cn=project-analytics [18:22:23] right? [18:27:04] ottomata: yes [18:27:41] so using project-analytics for file group ownership is the way to go, ja? [18:27:53] if i shouldn't use ou=projects? [18:28:16] all groups you'd want to use are in ou=groups [18:28:44] remember, though, that any group that starts with project- is managed by OpenStackManager [18:29:44] ok [18:29:56] that's ok though, right? those are the ones that we'd manage with labsconsole? [18:30:33] the goal here is to manage sensitive file access for users without shell accounts [18:31:09] labsconsole seems to be the easiest, since there is a web interfaces for adding ldap users into groups [18:31:20] if there's a better way we could do it [18:32:41] manage sensitive files? [18:32:45] that should likely go through ops [18:33:00] which means a group that isn't managed by OSM [18:33:07] (like ops, or wmf groups are now) [18:33:26] it's possible we'll also manage them through the web interface at some point, but it'll be managed differently [18:33:39] management of project- groups is *really* permissive [18:34:14] management of the more privileged groups will require two-factor auth when I add that feature [18:36:33] !log mw1161-mw1188 being pushed into pybal pool as general apaches [18:36:34] Logged the message, RobH [18:39:30] !log disregard, pushing one to test, mw1161 [18:39:31] Logged the message, RobH [18:47:58] RobH: mw1161 has been bitching about rsync, fwiw. [18:49:35] i dunno why its failing to sync, grrr [18:50:11] mw1161: rsync: change_dir#3 "/apache/common-local" failed: No such file or directory (2) [18:50:11] mw1161: rsync error: errors selecting input/output files, dirs (code 3) at main.c(643) [Receiver=3.0.9] [18:50:15] ^ That's what I got earlier today [18:51:52] ori-l: ping? [18:52:59] wikibugz: thx for info, seems to be 1161 specific [18:53:06] as the rest of my batch of new stuff is coming online now [18:59:11] Change merged: Ryan Lane; [operations/debs/gerrit] (master) - https://gerrit.wikimedia.org/r/47806 [19:04:55] (Ryan_Lane, sorry, was eating lunch, let's talk more about ldap after ops meeting) [19:05:02] ok [19:07:34] ori-l, re security review meeting, come on by! [19:08:13] just added you as a guest in google calendar [19:18:22] jeremyb_: can't help with this one. 11:25 -ChanServ(ChanServ@services.)- You have no special access to #wikimedia-tech. [19:18:43] and IRC people did not want Bugzilla tickets.. so shrug [19:19:06] i don't follow on the bugzilla+IRC [19:19:16] but i was just trying to tell you that you didn't need to leave [19:19:25] you can rejoin. leaving didn't help [19:19:50] it's fixed now [19:26:19] hi all! [19:26:53] seems like a stale copy of the Sites list used by Wikidata is stuck in memcached. [19:27:00] what's the procedure for purging key X from all memcached instances? [19:27:06] Is there a bug? [19:27:16] I mean a bug filed in Bugzilla. [19:27:25] i don't think there is one yet, no [19:27:56] I think any shell user purging keys from memcached will like one to track the issue. [19:28:03] <^demon> DanielK_WMDE: What's the key? [19:28:25] * aude can get it  [19:28:32] one sec [19:28:58] ^demon: good question... [19:29:08] sites/SiteList#2013-01-23+Site:2013-01-23 [19:29:17] is deployment still planned? [19:29:45] thanks aude [19:29:47] * DanielK_WMDE is slow [19:29:48] it's not a great solution but you could also try backporting my patch that changes the cache key [19:29:58] https://gerrit.wikimedia.org/r/#/c/47880/ [19:30:07] this shouldn't be the normal procedure for this [19:30:31] I'll submit a patch that sets a sane timeout for this [19:30:36] aude: when was the database updated? [19:30:37] good [19:30:47] friday [19:31:06] <^demon> aude: Where's the key defined? I can't find it :\ [19:31:28] it's in SiteSQLStore.php [19:31:33] and uses constant here: [19:31:34] https://gerrit.wikimedia.org/r/#/c/47880/1/includes/site/SiteList.php [19:31:57] to help enable us to refresh things if we change serialization format or some breaking change in the code itself [19:32:53] ok, so SpecialSupportedLanguages has one hour hard coded. I guess we can go with that. [19:33:40] aude: oh, it's SiteList#... but Site:..., I guess I didn't pay attention there, how ugly :P [19:34:00] aude: next time you change the cache key, maybe take the opportunity to make that consistent [19:34:00] heh [19:34:11] sure [19:35:16] <^demon> deleted wikidatawiki:sites/SiteList#2013-01-23+Site:2013-01-23 [19:35:24] cool! [19:35:44] <^demon> I forgot the prefix :) [19:35:47] <^demon> Originally. [19:36:02] <^demon> Had to echo wfMemcKey(...) from eval.php :) [19:36:02] right [19:36:48] ^demon: so, how does that work? Is there a script that goes through all memcached instances? [19:37:36] <^demon> maintenance/mcc.php [19:37:47] <^demon> > delete [19:38:01] <^demon> mcc.php is a fun toy :) [19:38:05] still see no minwiki [19:41:10] no minwiki in api.php (it gets cached?), no minwiki in wb.getSites() [19:41:13] :( [19:41:36] o_O [19:41:39] wtf? [19:42:14] aude: is the current code old and using a different key? [19:43:03] DanielK_WMDE: it is [19:43:15] it has jan 23 cache version key [19:43:56] i suggest my patch, at least this time [19:44:03] give it a new cache key [19:44:28] yea... [19:44:46] hm, i don't see that commit in any deployment branch? [19:44:59] the new key was added in f556c7f2ae3dc09aa55801d42b8002946b768f92 [19:45:01] <^demon> Cherry pick it to 1.21wmf9, and we can deploy. [19:45:04] whouldn't that be in wmf12? [19:45:18] er, wmf9 [19:45:36] aude: just cherry-picking the patch would also change the key [19:45:56] sure [19:46:01] wmf9 [19:46:02] ^demon: i suggest to merge & backport one more patch: https://gerrit.wikimedia.org/r/#/c/48490/ [19:46:25] i can submit it to gerrit [19:46:28] Change merged: Ottomata; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48249 [19:46:35] oh, that's yours [19:46:35] <^demon> Yeah, that change looks good too. [19:46:38] that would be a godo one [19:46:42] good [19:49:38] <^demon> brb. [19:50:11] Ryan_Lane: the mobile team is expecting to push login functionality into the stable version of MobileFrontend in production tomorrow. at the moment (in beta, limited usership) we are forcing users who log in to use HTTPs. would it be problematic if we continue to force https for users who log in in production (we can easily disable the forcing of HTTPS with a config var) [19:50:13] https://gerrit.wikimedia.org/r/#/c/48492/ for updating the cache key and daniels patch would be good too [19:50:17] ^demon: no hurry [19:50:41] awjr: you mean once they are logged in they continue to use https? [19:50:49] Ryan_Lane: correct [19:51:02] I'd prefer that, in fact [19:51:09] cool :) [19:51:12] as long as when they log out they default back to http [19:51:27] are you using the same code as desktop to handle this? [19:51:39] with the cookie not marked as secure? [19:51:40] Ryan_Lane: you mean the stick https cookie? [19:51:43] yes [19:51:44] yes [19:51:44] <^demon> aude: Ok back [19:51:45] cool [19:51:51] * aude still here [19:52:23] <^demon> Ok, +2'd 48942. Did you backport Daniel's fix too? [19:52:29] Ryan_Lane: ok, awesome. i'll dbl check the logout funcionality (i know it kills the cookies but it might keep on you https otherwise) [19:52:40] ^demon: i think someone has to merge that one [19:52:46] * aude can't +2 [19:52:56] <^demon> Oh, well let's do that. [19:53:03] https://gerrit.wikimedia.org/r/#/c/48490/ [19:53:09] looks good to me :) [19:53:10] <^demon> Already done. [19:53:25] ok, and then we can cherry pick [19:53:36] * DanielK_WMDE wibbles [19:53:58] <^demon> After all this cherry picking, I wouldn't mind some pie. [19:54:21] ^demon: uh, it's not merged? [19:54:34] ^demon: you should have come to the office at wmde today. four birthdays. [19:54:34] <^demon> waiting on jenkins to run tests and merge. [19:54:40] one of them our own aude here [19:54:41] ah, sorry [19:54:42] <^demon> I +3'd. [19:54:43] (waves) [19:54:43] <^demon> +2. [19:54:45] <^demon> Wow, no +3. [19:54:50] heh [19:54:52] * aude eating cakes [19:54:57] more than just one pie ;) [19:55:28] hi hashar [19:56:07] jenkins is slow today [19:56:28] maybe that's why hashar keeps leaving... he didn't want to hear that :P [19:56:33] heh [19:57:03] DanielK_WMDE: na my IRC client sent me a notification about your message, before it finished joining all channels. [19:57:15] this should be hard enough kick for memcached to refresh [19:57:17] DanielK_WMDE: seems that produced some kind of race condition that made the client to skyrocket its CPU usage :-] [19:57:25] :o [19:57:30] short history: you killed my client you bastard! ;-]]]]]]]]]]]]]] [19:57:35] hashar: hahaha! [19:57:45] paravoid: hey, what's up? [19:57:47] DanielK_WMDE: so that is Wikidata on enwiki tonight ? [19:57:52] ottomata1: cool; thanks for the invite! is it happening today? [19:57:59] remember kids: when hashar joins, greet him as quickly as possible :P [19:58:13] naw tomorrow [19:58:19] hashar: if Reedy shows up, yea. Or maybe ^demon wants to do it... [19:58:39] * DanielK_WMDE goes and does the ESTA thing now. [19:58:42] <^demon> DanielK_WMDE: I've already said I'll do it if we can't track down Reedy :) [19:58:43] * aude wants to go around removing site links on enwiki :D [19:58:48] but no hurry [19:58:49] I would do it if I knew how to do it :-] [19:58:56] I guess I should receive some training from Sam [19:59:00] paravoid: can i pm? [19:59:06] it's actually just a few steps [19:59:28] merging my patches for settings and wikibase submodule change for wmf9 [19:59:28] Ryan_Lane, back with the ldap Qs [19:59:34] then switch enwiki to wmf9 [19:59:57] it might be wise to poke around on test2 wikiepdia first once the settings and submodule get merged into wmf9 [19:59:57] we were talking about sensitive file access [19:59:59] in hdfs [20:00:18] the plan was to use a Labs project, but you think we should do something else? [20:00:24] <^demon> aude: Did you have an incoming cherry pick for the cache expiry? [20:00:27] * aude doesn't know if git deploy is being used or scap, though [20:00:40] * aude waiting on jenkins which appears done now [20:00:41] scap [20:01:14] and I have no idea what git-deploy status is [20:03:09] ^demon: https://gerrit.wikimedia.org/r/#/c/48493/ [20:03:27] well, i think scap is slightly easier with updating just enwiki [20:03:45] it takes a very long time on a new branch to rebuild entire localisation cache etc. [20:04:10] * aude needs to learn more about git-deploy [20:06:08] <^demon> aude: Jenkins is doing its thing. [20:06:31] ok [20:11:07] !log demon synchronized php-1.21wmf9/includes/site 'Updating sites code to pull in I8964a03c, I42221832' [20:11:08] Logged the message, Master [20:11:46] <^demon> Ok, sites fixes all live. [20:12:57] <^demon> No minwiki, but hahaha, I think I might've figured it out. Is there something other than the action=help where we can view the supported wikis? [20:13:50] javascript [20:13:53] wb.getSites() [20:14:19] * aude does hard refresh [20:14:21] that might be fudged by cached JS modules, though [20:14:31] <^demon> Well, action=help is *definitely* cached. [20:14:33] <^demon> I totally forgot that. [20:14:35] i don't think we have a way yet to force squids to purge that [20:14:38] <^demon> Until about a minute ago. [20:14:48] it's there! [20:14:52] \o/ [20:15:07] <^demon> Yay. [20:15:31] http://www.wikidata.org/w/index.php?title=Q5296&diff=6302854&oldid=6204397 [20:15:32] but we may still have stale JS modules on the squids. oh, well. [20:15:35] very cool! [20:15:40] yes we might [20:16:23] thanks ^demon [20:16:38] <^demon> np. [20:16:40] <^demon> What's next? [20:16:52] yes, the community is happy to have minwiki in wikidata [20:16:53] wmf9 + wikidata client on enwiki? :) [20:17:11] * aude goes to move but 44750 to done [20:17:47] err, bug 44750 [20:19:40] <^demon> RobH: I'm getting the same rsync error on mw1165 now too. [20:21:01] New patchset: Demon; "enwiki to 1.21wmf9" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48518 [20:21:37] Change merged: Demon; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48518 [20:21:40] yay! [20:22:10] although it's no wikidata quite yet [20:22:31] we need https://gerrit.wikimedia.org/r/#/c/48465/ [20:22:33] <^demon> One step at a time :) [20:22:34] !log demon rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf9 [20:22:35] Logged the message, Master [20:22:44] then https://gerrit.wikimedia.org/r/#/c/48461/ [20:22:50] sure [20:25:12] gosh, is it just me or enwiki is slow now? [20:25:26] <^demon> Crap. [20:25:29] <^demon> itwiki needs 1.21wmf9. [20:25:31] <^demon> Fatal error: Call to a member function getNavigationIds() on a non-object at /usr/local/apache/common-local/php-1.21wmf8/extensions/Wikibase/client/includes/LangLinkHandler.php on line 333 [20:25:43] ugh? [20:25:45] <^demon> And he. [20:25:51] <^demon> And any other client on 1.21wmf8. [20:25:58] huh [20:26:18] <^demon> That error is spamming fatal.log for itwiki and hewiki. [20:26:23] ugh [20:27:17] ah darn [20:27:37] how just now? [20:27:47] !log demon rebuilt wikiversions.cdb and synchronized wikiversions files: hewiki and itwiki to 1.21wmf9 [20:27:48] Logged the message, Master [20:28:03] i assume demon update wikibase to wmf9 [20:28:13] but not huwiki? [20:28:21] <^demon> I was only seeing it on hewiki and itwiki. [20:28:27] probably hu had the same issue but it looks like he? [20:28:27] * aude confused how an error just appeared now [20:28:32] not for the past weeks? [20:28:35] ottomata: hey, join me in #wikimedia-tech? [20:29:12] <^demon> aude: I have no idea... [20:29:21] <^demon> Could've been going for weeks for all I know. [20:29:33] <^demon> Logs got very quiet after I sync'd. [20:29:38] :( [20:29:44] uh oh, i better get off of RT duty in the topic [20:29:46] notpeter! [20:29:47] <^demon> Ah, there's huwiki, [20:29:50] save me! [20:29:51] huh, $site->getNavigationIds(); [20:29:52] hehe :) [20:29:57] i wonder which site? [20:29:59] minwiki? [20:30:06] err. line 333 of LangLinkHandler is blank for me on the mw1.21-wmf9 branch [20:30:09] no [20:30:15] DanielK_WMDE: wmf8 [20:30:18] ah [20:30:37] !log demon rebuilt wikiversions.cdb and synchronized wikiversions files: huwiki and itwiki to 1.21wmf9 [20:30:38] Logged the message, Master [20:31:31] i can make a quick patch to avoid the fatal error [20:31:44] that would be good [20:31:45] no idea about the root cause yet. looks like sites object again. [20:31:47] drat. [20:31:47] New patchset: Demon; "Moving itwiki, huwiki and hewiki to 1.21wmf9" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48559 [20:31:49] for wmf9, i guess [20:31:56] Change merged: Demon; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48559 [20:31:56] if $site is false? [20:32:08] yes. if (!$site) continue; [20:32:10] basically [20:32:12] and some logging [20:32:16] sites->getSite( $wiki ) wiill be false if it cant' find one [20:32:20] such as minwiki, perhaps [20:32:27] exactly [20:32:38] that shouldn't cause a fatal error [20:32:41] that would explain if it happened since the weekend or something [20:32:42] needs to be more robust [20:32:42] right [20:32:47] I take it wikidata isn't wmf8 compatible anymore? [20:32:48] yeah [20:32:56] no, since the sites object was cached, remember? [20:33:01] it's only now using the new version [20:33:03] <^demon> robla: It would seem not ;-) [20:33:08] robla: ah yes [20:33:09] so now it's hitting the issue [20:33:23] <^demon> robla: I moved huwiki, hewiki and itwiki to 1.21wmf9 as well, fatal.log was exploding. [20:33:35] robla: i don't think it's a compat issue. [20:33:44] we can add more checks for this sort of thing in the code [20:33:49] it's just brittle against inconsistencies in the sites table between repo and client [20:33:58] and , of course, let's wait and see before switching on enwiki to make sure it's okay [20:34:03] New patchset: Tpt; "(bug 44032) Deploy Universal Language Selector to oldwikiource" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47732 [20:34:06] with a patch from daniel, i'd say [20:34:09] ^demon robla fwiw, VisualEditor seems to be hanging trying to load pages on enwiki right now, progress bar spins forever looks like [20:34:16] ^demon: you purged the cached thingy from memcached, using the repo's key, right? [20:34:28] i think the clients need purging too [20:34:28] ^demon: do the same using the client wiki's key prefixes, please [20:34:33] yes [20:34:36] aude: exactly [20:34:45] <^demon> Will do, sec. [20:34:52] thanks [20:34:58] James_F|Away: you know about the VE problem chrismcmahon just mentioned? ^ [20:34:58] * DanielK_WMDE goes to code the patch [20:35:01] <^demon> chrismcmahon: Yeah, I saw. Fighting other fires first. [20:35:07] <^demon> robla: Happend on 1.21wmf9 update. [20:35:34] robla: Gah. Looks like Parsoid is not responding. [20:35:45] robla: I'll check after meeting. :-( [20:36:09] New patchset: Tpt; "(bug 44032) Deploy Universal Language Selector to oldwikiource" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/47732 [20:36:09] James_F|Away: seems to work: http://cerium.wikimedia.org:6081/mw/Parsoid [20:36:24] Hmm. [20:36:28] James_F|Away: did you change visual editor so the edit links use the normal edit system? [20:36:48] aude: Yes, about two months ago; did that code finally get deployed? [20:36:54] gah. [20:36:56] I really have to go. [20:36:57] James_F|Away: ok, then that's fine [20:36:57] Bye. [20:37:16] aude: i'm a bit confused about versions. the patch should go against what, wmf9 and master? [20:37:38] <^demon> DanielK_WMDE: keys cleared. [20:37:53] wmf9 [20:37:57] master would be good too [20:38:14] ^demon: please look if the error is still ocurring. [20:38:14] ^demon: let's see if that helps, although a patch for the code is good too [20:38:24] oh, and: are we sure that the sites table is consistent on all the wikis? [20:38:37] Reedy did update them, i believe [20:38:40] yea, we want that check in any case. [20:38:41] <^demon> Well, the error went away when I moved the wikis to 1.21wmf9. [20:38:44] but not sure about *all* the wikis [20:40:05] * aude assumes the toolserver just doesn't have the views added for sites table on itwiki, huwiki, etc. [20:40:11] sure the tables are there but can't see them [20:40:28] ^demon: good [20:40:50] good [20:41:20] let's make sure enwiki has an updated sites table too [20:41:36] <^demon> I haven't verified all the rows, but they all (he, hu, it) have 790 rows. [20:41:50] select * from sites where site_global_key = 'minwiki'; [20:41:56] 790 is a good number [20:41:59] <^demon> ERROR 1146 (42S02): Table 'enwiki.sites' doesn't exist [20:42:04] oh, really? [20:42:06] ugh [20:42:28] well, the schema patch for that is in core [20:43:05] then mwscript extensions/Wikibase/lib/maintenance/populateSitesTable.php [20:43:12] (todo, move the script to core) [20:45:25] New patchset: Hashar; "contint: php-luasandbox to latest version" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48127 [20:45:41] i suggest to copy the sites table from wikidatawiki instead of populating it with the script [20:45:47] site_identifiers, too [20:45:56] that works [20:46:00] either way [20:46:57] Change abandoned: Hashar; "I will let ops handle the cleanout, I got too many changes pending review already." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/44964 [20:54:01] aude, ^demon: gah, coded against a stale branch. fiddling it in now. [20:54:20] <^demon> The heck. [20:55:12] 'nother problem? [20:55:19] ok [20:57:09] binasher: have you looked at zipkin? http://twitter.github.com/zipkin [20:57:23] distributed, full stack performance tracing [20:57:32] <^demon> Denny_WMDE: Not wikidata's fault, my fault :) [20:57:41] hmmm, ok [20:58:19] ok, there are my wikidata changes in itwiki [20:58:24] * aude switches on enhanced changes [20:58:54] oh, never mind [20:59:02] they have wmf9 but not the patches for that yet [20:59:13] we didn't update the submodule with the fixes yet [20:59:17] dschoon: yeah, i've been doing a lot of talking to people at twitter lately and have seen some live visualization of data derived from it in their office. definitely cool! [20:59:24] nice. [20:59:31] but otherwise everything still seems fine to me [21:00:07] there is a bug in the preferences page (fixed when we update the submodule), it shows "wbc-rc-show-wikidata-pref" [21:00:28] ooops, wrong channel [21:00:49] when we update the submodules, then the wikidata clients will get a few fixes like fixing the message key [21:00:53] binasher: it seems like it'd be non-trivial to integrate, as i don't believe the php support for thrift or zookeeper is all that strong. [21:01:17] (thrift due to scribe) [21:01:29] our mediawiki profiling (just as we currently collect it) unfortunately doesn't offer a way to view requests as a whole, each part of the trace goes into its own untracked bucket [21:01:29] my mistake, zk isn't required except for script. [21:01:34] *nod* [21:01:52] but i think some of its nicer features can be implemented with incremental development of what we have [21:01:56] i'd be interested in a braindump next time you're in the office. if you've talked to them, you surely have a better idea of the reqs. [21:02:00] nice. [21:02:01] itwiki looks fine to me [21:02:22] there's also a hope for deprecating our use of mediawiki's own profiler for xhprof [21:02:28] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48461 [21:03:17] !log demon synchronized wmf-config/CommonSettings.php 'wikidata client on enwiki' [21:03:18] Logged the message, Master [21:03:20] that's facebook, yes? [21:03:41] !log demon synchronized wmf-config/InitialiseSettings.php 'wikidata client on enwiki' [21:03:42] Logged the message, Master [21:04:35] PHP fatal error in /usr/local/apache/common-local/php-1.21wmf9/extensions/Wikibase/client/includes/LangLinkHandler.php line 367: [21:04:35] Call to a member function getNavigationIds() on a non-object [21:04:52] !log auth-dns update [21:04:53] Logged the message, Master [21:05:08] <^demon> Denny_WMDE: Where? [21:05:13] https://en.wikipedia.org/wiki/Wikidata [21:05:24] aude, ^demon: ugh, finding more places where missing site entries cause fatal errors [21:05:36] dschoon: yeah, xhprof is from fb [21:05:46] Denny_WMDE: working on a patch [21:06:00] can we update localisation for wikibase? [21:06:05] ^demon: [21:06:19] i see wbc-rc-show-wikidata-pref but believe it's fixed in the code and may require localisation update [21:06:31] DanielK_WMDE: good to fix them, then [21:06:31] <^demon> Let's not run l10n update until we're sure nothing else is exploding. [21:06:37] ^demon: ok [21:06:40] this is not critical [21:06:52] <^demon> You could make the message on-wiki to workaround possibly? [21:06:59] ah, yes [21:07:02] * aude doing [21:08:23] was the sites table created and copied? [21:08:26] wtf. getSite() returns null for unknown sites. [21:08:33] the code checks === false in several places. [21:08:33] <^demon> Denny_WMDE: Yup. [21:08:34] ugh. [21:08:36] !log demon synchronized wmf-config/InitialiseSettings.php 'Wikibase client throwing fatals, disabling on enwiki pending fix' [21:08:37] Logged the message, Master [21:08:38] anyway, i was able to adde "does not work yet with enhanced changes" in the preferences page [21:08:48] that's not in the code itself, but nice to have [21:08:50] and this is in core :/ [21:09:00] thanks for disabling, ^demon [21:09:26] ottomata: no. I said we should *not* use a project for sensitive file access [21:09:27] let's get a patch in [21:09:56] right, just bringing the subject back up [21:10:02] reiterating where we left off [21:10:20] ottomata: thanks for updating topic! [21:10:22] so if we don't use a project, what should we do? manual group add in ldap? [21:10:34] notpeter: thanks for taking over!) [21:10:44] ^demon: enwiki has sites table now, right [21:10:49] <^demon> Yup. [21:10:49] Ryan_Lane^^ [21:10:52] ok [21:11:02] <^demon> 790 rows. [21:11:04] ottomata: yes [21:11:05] good [21:11:10] and ops should manage it [21:11:47] <^demon> aude: And 1580 in site identifiers, as expected. [21:11:53] site_identifiers? [21:11:54] ok [21:12:08] <^demon> Yeah, I'm taking shortcuts since we're all busy. [21:12:14] <^demon> Things like underscores may be omitted :) [21:12:18] ok, that is fine [21:15:04] Ryan_Lane, that's cool, do we currently have a process for manual ldap groups? is there anyone else doing that yet? [21:15:12] yes [21:15:18] for both the ops and wmf groups [21:15:23] put a ticket into rt [21:15:48] to do what? create new NDA specific group(s) in ldap? [21:16:04] I mean when there's changes in membership needed [21:16:09] ohoh [21:16:11] right, cool [21:16:27] what's the ops group? [21:16:31] (wmf == staff?) [21:16:36] yes [21:16:38] and ops = ops [21:16:39] ops == ops team? i guess i mean, what's ops used for? [21:18:17] yes [21:19:10] jeremyb_: Your access to RT is all set up. You should have a copy of the RT ticket headed your way with more details, but otherwise you are good to go, just have to use the password reset tool. [21:19:19] I used the email account that was already on ticket for the RT account. [21:19:39] <^demon> aude: I think I know what did it. We turned on wikibase before the table was done populating, so we probably have a bad cache entry. Will flush for enwiki. [21:19:48] oh [21:19:58] * aude poking around on the other wikipedias [21:20:22] ok, so Ryan_Lane, what's next then, I'll update the existing RT with the conclusion of this conversation (create manual ldap groups for NDA level access, don't use labs projects for file access) [21:20:44] well, it's fine to use labs projects for file access [21:20:48] but not for sensitive file access [21:20:51] can you and I just make the decision as to what these groups are named (i only care about one right now, for the data NDA), or do we need to get more people into the discussion [21:21:05] right now, that's the only access I need to manage in hdfs [21:21:08] access to webrequest logs [21:21:12] ok [21:21:27] ARGH [21:21:38] <^demon> aude: To be safe, let's wait until we've got those safeguards in place :) [21:21:39] What the hell did I miss to deploy apaches, why arent they being fed a load.... [21:21:41] I think naming the group nda is fine [21:22:04] yes, please [21:22:05] RobH: are they in the pybal config? [21:22:14] they are, and i see them in ipvsadm [21:22:16] and they are pooled [21:22:21] but they dont have any active connections [21:22:22] ok, thanks, I'll update that ticket [21:22:32] RobH: how do you know they aren't getting traffic? [21:22:38] and the inactive connection hasnt raised up since before lunch [21:22:45] and have no graphing spikes in ganglia [21:22:56] ah [21:22:57] ok [21:23:06] mostly know they arent being used due to the inactcon count not rising at all [21:23:28] so they have the doc root (well, two dont but i disabled them) [21:23:31] RobH: are they being depooled? [21:23:31] RobH: how did you know i was just getting ready to open my laptop again? :) [21:23:33] and show ok in pybal. [21:23:34] check the log [21:23:39] Ryan_Lane: Pybal says pooled [21:23:45] * jeremyb_ scrolls up [21:23:51] RobH: check pybal's log to see if they are being pooled/depooled [21:24:12] i did, they are pooled, not being depooled [21:24:15] example mw1162 [21:24:27] i show enabled/up/pooled [21:24:32] yet no active connections. [21:24:46] RobH: no password reset required. didn't even have to log in (was already logged in from earlier today). it Just Works. [21:25:59] Ryan_Lane: has the lo:lvs bound [21:26:04] Ryan_Lane, ok then, to get this moving, shall I gather up a list of all the related Data NDA signers and create an access-request RT? [21:26:13] ottomata: yes [21:26:16] cool, thanks [21:26:18] ottomata: good luck [21:26:37] haha [21:27:06] jeremyb_: awesome [21:27:13] grrr, why doesn't RT autolinkify URLs? [21:28:28] eventually the new servers hsould get a slightly higher weight, as they are beefier than the old apaches [21:28:36] but not until i know why they are getting no traffic =/ [21:28:56] pybal sees and pools, has the lo lvs interface bound, in all node groups and such [21:31:00] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48439 [21:34:17] New patchset: RobH; "left out mw1199 in regex by mistake" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48566 [21:34:28] DanielK_WMDE: what was the reason two explicit BEGINs only gives a warning? [21:35:49] New review: RobH; "spoon" [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/48566 [21:35:50] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48566 [21:36:42] New review: Andrew Bogott; "I think I'm happy for image uploads to be disabled by default on new installs -- so, fine to just ha..." [operations/puppet] (production); V: 2 C: 2; - https://gerrit.wikimedia.org/r/48441 [21:36:44] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48441 [21:36:59] im merging someone's change about nuke to Nuke on sockpuppet [21:37:06] in mediawiki.pp [21:37:31] andrewbogott: ^ [21:37:33] your change [21:37:50] i imagine was fine, since you merged in gerrit, but it is now live on cluster [21:37:51] RobH: Yep, go ahead and merge. [21:37:56] cool [21:37:58] its live [21:38:06] I'm going through a set of patches, just hadn't made it to the end yet. [21:41:15] Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47585 [21:49:22] ^demon: I added the new version of gerrit to the repo [21:49:30] <^demon> Cool, I'll test out labs. [21:49:30] <^demon> Thanks. [21:49:33] yw [21:52:01] AaronSchulz: legacy code causing fatal errors otherwise. [21:52:23] I'd be happy with a more aggressive approach, as long as I'm not skinned for it in the end :P [21:55:51] ^demon: https://gerrit.wikimedia.org/r/#/c/48568/ [22:03:03] !log demon synchronized php-1.21wmf9/extensions/VisualEditor/modules/ve/init/mw/ve.init.mw.Platform.js 'Touching file' [22:03:04] Logged the message, Master [22:04:42] hey notpeter [22:05:22] * jeremyb_ has some RT questions [22:06:34] paravoid: Do you have a moment to look over my new apache deployment of servers and see why they aren't getting a load? I think it is a pybal issue, and it needs to be restarted, but not sure. [22:06:48] its showiing in the configuration file, and shows as pooled in the logs. [22:06:54] <^demon> Ryan_Lane: update worked fine on labs. [22:06:59] notpeter took a look and we both dunno WTF is up with it [22:07:00] great [22:07:08] <^demon> We should be set for this evening then \o/ [22:07:10] before i go kicking pybal i want soemone else to look if possible. [22:07:24] * RobH pings mark in hope he may be about [22:07:25] heh [22:09:31] New patchset: Demon; "Disabling wikibase client on enwiki until we redeploy" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48571 [22:10:38] New review: Demon; "This is already live. We'll probably re-deploy tomorrow." [operations/mediawiki-config] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/48571 [22:10:50] RobH: hey [22:10:51] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/48571 [22:11:03] which servers are those? [22:12:30] New patchset: Cmjohnson; "adding mw1201-1220 fixing mw1161-1200 (not in alphabetical order)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48572 [22:13:24] Change merged: Cmjohnson; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48572 [22:34:38] New patchset: Demon; "Adjust apache proxy to treat gerrit requests exactly as given" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48574 [22:34:47] <^demon> Ryan_Lane: Whoops, I totally forgot to commit ^ earlier. [22:35:20] is this something you'd want me to merge now, or later? [22:35:53] <^demon> I'm pretty sure it'll be fine now, but it'll definitely need to be merged before we try to bring the new gerrit up. [22:35:57] <^demon> Maybe merge while it's down? [22:36:12] ok [22:40:08] New review: Faidon; "Much, much better than before, great! A few comments:" [operations/puppet] (production); V: 0 C: -1; - https://gerrit.wikimedia.org/r/48041 [22:41:33] jeremyb_: what's up? [22:41:50] The sky, unless you're a bat [22:42:02] notpeter: some RT questions [22:42:27] notpeter: i was pointed to you for RT duty [22:43:38] * ^demon whacks Damianz [22:43:49] * Damianz enjoys it [22:50:28] New review: Ottomata; "- What does kraken mean?" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/48041 [22:55:22] paged? [22:55:49] yeah.... [22:55:51] lvs ipv4 appservers in equiad [22:56:00] wtf nagios-wm [22:56:06] is someone awake looking at this? [22:56:11] apergos: yeah [22:56:14] thanks [22:57:09] 1am per my back of the napkin calculations [22:57:22] exactly right [23:02:10] <^demon> Ryan_Lane: I'm gonna dip out and grab dinner before we get started [23:02:18] ^demon|away: sounds goo [23:02:20] *good [23:08:24] no route to host on appservers check now... neat. [23:12:29] nagios-wm is borked due to a config issue on its startup. [23:12:37] atleast per the wikitech docs on starting it. [23:18:46] notpeter: so paravoid noticed why [23:18:55] the lvs servers hvae NOT had their run to row C done yet [23:19:02] or the new network ip for row C bound to the interface [23:19:08] So it has no way to actually send traffic to them. [23:19:15] how awesome is that? [23:19:27] !log unpooled mw1161-1188 from pybal config [23:19:28] Logged the message, RobH [23:19:53] woo! [23:19:55] awesome. [23:21:13] at least I know why [23:21:16] it was breaking my brain. [23:21:29] 'WHAT AM I MISSING ARGHHHH' [23:21:31] bleh. [23:28:26] New patchset: Pyoungmeister; "correcting thankfully not really disasterous typo in coredb.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48579 [23:30:02] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/48579 [23:39:27] <^demon|away> Ryan_Lane: I'm still dinnering, but fyi: http://etherpad.wmflabs.org/pad/p/GerritUpgrade [23:39:32] PROBLEM - Puppet freshness on analytics1007 is CRITICAL: Puppet has not run in the last 10 hours [23:43:38] ^demon|away: bed time for me, enjoy the upgrade! [23:43:53] <^demon|away> Goodnight, and thanks for your help with zuul. [23:44:00] I will reply to your wikitech-l announce with round of beers :-] [23:44:05] wel [23:44:15] zuul is just the tiniest part of all [23:44:25] :-D [23:44:54] sleeping for now, [23:45:54] yeah so [23:46:00] zuul might reconnect by itself :-] [23:46:01] but maybe not [23:46:10] you will see, not a big deal anyway [23:46:23] have fun! [23:46:26] * hashar waves [23:51:42] LeslieCarr: you should help these folks out: http://sfbay.craigslist.org/sfc/cpg/3606743947.html [23:55:17] AaronSchulz: hey, im updating wikitech memcached docs [23:55:25] in the old setup, down servers needed to be pulled out of mc.php [23:55:32] on the new mc-site.php, does the same hold true? [23:55:39] (its no longer using old slot format, so not sure) [23:56:01] One would hope its smart enough these days to see a server is down and just not use it, but i have no idea. [23:56:16] (ryan/patrick suggested you may know) [23:57:30] ori-l: ping [23:57:39] preilly: what's up? [23:58:04] ori-l: are you available at all to chat? [23:58:39] sure [23:58:50] preilly: i need coffee badly, want to step out for a few? [23:59:08] ori-l: yeah [23:59:21] ori-l: I'll meet you in the lobby in 5 (Does that work?) [23:59:32] works.