[00:00:06] PROBLEM - check_raid on silicon is CRITICAL: CRITICAL md0 status=[UU]. md1 status=[UU]. md2 status=[UU]. md3 status=[U_]. [00:00:49] ori: I think I know where the problem comes from... do you do git fetch before pulling in /a/common ? [00:01:09] yes [00:01:19] mh [00:01:20] weird [00:05:06] PROBLEM - check_raid on silicon is CRITICAL: CRITICAL md0 status=[UU]. md1 status=[UU]. md2 status=[UU]. md3 status=[U_]. [00:10:06] PROBLEM - check_raid on silicon is CRITICAL: CRITICAL md0 status=[UU]. md1 status=[UU]. md2 status=[UU]. md3 status=[U_]. [00:15:06] PROBLEM - check_raid on silicon is CRITICAL: CRITICAL md0 status=[UU]. md1 status=[UU]. md2 status=[UU]. md3 status=[U_]. [00:16:30] (03PS1) 10Faidon Liambotis: ldap: switch CA name to Equifax_Secure_CA [operations/puppet] - 10https://gerrit.wikimedia.org/r/121283 [00:16:43] (03CR) 10Faidon Liambotis: [C: 032 V: 032] ldap: switch CA name to Equifax_Secure_CA [operations/puppet] - 10https://gerrit.wikimedia.org/r/121283 (owner: 10Faidon Liambotis) [00:19:15] mh.. I guess I'm going to investigate on the git hook stuff after my next deploy [00:20:10] PROBLEM - check_raid on silicon is CRITICAL: CRITICAL md0 status=[UU]. md1 status=[UU]. md2 status=[UU]. md3 status=[U_]. [00:20:20] RECOVERY - Certificate expiration on virt0 is OK: (null) [00:20:29] \o/ [00:20:52] springle: is es3 decommissioned? [00:23:10] RECOVERY - Puppet freshness on labstore2 is OK: puppet ran at Thu Mar 27 00:22:53 UTC 2014 [00:25:10] PROBLEM - check_raid on silicon is CRITICAL: CRITICAL md0 status=[UU]. md1 status=[UU]. md2 status=[UU]. md3 status=[U_]. [00:26:12] RECOVERY - Certificate expiration on virt1000 is OK: (null) [00:30:12] PROBLEM - check_raid on silicon is CRITICAL: CRITICAL md0 status=[UU]. md1 status=[UU]. md2 status=[UU]. md3 status=[U_]. [00:31:32] !log acknowledge silicon's check raid in icinga, already on rt7136 [00:31:38] Logged the message, RobH [00:32:32] paravoid: Got maybe time for a small ci change that would make my day? https://gerrit.wikimedia.org/r/#/c/119750/ [00:35:17] PROBLEM - Certificate expiration on virt1000 is CRITICAL: SSL error: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed [00:38:22] (03CR) 10Faidon Liambotis: [C: 04-1] contint: override .jshintrc file on gallium (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119750 (owner: 10Hashar) [01:03:30] RECOVERY - Certificate expiration on virt1000 is OK: (null) [01:29:24] paravoid: yes https://rt.wikimedia.org/Ticket/Display.html?id=6544 [01:36:27] !log Reloading Zuul to deploy I8a7ccef26da45d9ed7c7705df77246001bb85544 [01:36:36] Logged the message, Master [02:02:40] (03PS2) 10Jforrester: Fix typo in noncirrus.dblist reference [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120172 [02:03:13] (03PS3) 10Krinkle: Fix typo in noncirrus.dblist noc symlink creation [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120172 (owner: 10Jforrester) [02:03:17] (03CR) 10Krinkle: [C: 032] Fix typo in noncirrus.dblist noc symlink creation [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120172 (owner: 10Jforrester) [02:03:29] (03Merged) 10jenkins-bot: Fix typo in noncirrus.dblist noc symlink creation [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120172 (owner: 10Jforrester) [02:04:01] !log LocalisationUpdate failed (1.23wmf18) at 2014-03-27 02:04:01+00:00 [02:04:13] Logged the message, Master [02:04:37] !log LocalisationUpdate failed (1.23wmf19) at 2014-03-27 02:04:37+00:00 [02:04:44] Logged the message, Master [02:11:33] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Mar 27 02:11:30 UTC 2014 (duration 11m 29s) [02:11:39] Logged the message, Master [02:14:20] PROBLEM - Puppet freshness on tantalum is CRITICAL: Last successful Puppet run was Wed 26 Mar 2014 05:12:06 PM UTC [02:58:51] (03PS1) 10Chad: Opt all remaining wikis with ~50k or less pages into Cirrus as Beta [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121294 [03:01:27] (03CR) 10Chad: [C: 032] Opt all remaining wikis with ~50k or less pages into Cirrus as Beta [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121294 (owner: 10Chad) [03:01:35] (03Merged) 10jenkins-bot: Opt all remaining wikis with ~50k or less pages into Cirrus as Beta [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121294 (owner: 10Chad) [03:03:26] !log demon synchronized noncirrus.dblist [03:03:32] Logged the message, Master [03:04:22] !log demon synchronized wmf-config/InitialiseSettings.php 'touch' [03:04:28] Logged the message, Master [03:09:13] ^d: nice, congrats [03:10:41] <^d> :) [03:11:01] <^d> Only 85 wikis don't have Cirrus at all yet. [03:11:03] <^d> Almost there. [03:13:12] <^d> This is my favorite Elasticsearch graph right now. http://ganglia.wikimedia.org/latest/graph.php?r=week&z=xlarge&c=Elasticsearch+cluster+eqiad&m=cpu_report&s=by+name&mc=2&g=cpu_report [03:13:15] (03PS1) 10Springle: reassign db1034 to s7 [operations/puppet] - 10https://gerrit.wikimedia.org/r/121295 [03:13:21] <^d> Showing nice even daily traffic cycles finally. [03:13:31] <^d> (Minus when Nik or I mess with things :)) [03:14:54] (03CR) 10Springle: [C: 032] reassign db1034 to s7 [operations/puppet] - 10https://gerrit.wikimedia.org/r/121295 (owner: 10Springle) [03:18:52] !log xtrabackup clone db1007 to db1034 [03:18:57] Logged the message, Master [03:25:41] (03CR) 10Mattflaschen: "It's no longer blocked by that bug. I need to rebase and retest." [operations/puppet] - 10https://gerrit.wikimedia.org/r/79955 (owner: 10Mattflaschen) [03:34:55] (03PS1) 10Jalexander: Add config file for bing webmaster tools. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121297 [03:44:00] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:58:52] (03CR) 10Krinkle: contint: override .jshintrc file on gallium (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119750 (owner: 10Hashar) [04:00:00] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 307199 bytes in 8.144 second response time [04:02:50] PROBLEM - gitblit.wikimedia.org on antimony is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Temporarily Unavailable - 516 bytes in 0.026 second response time [04:04:00] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 308675 bytes in 9.080 second response time [04:09:28] poor gitblit [05:15:20] PROBLEM - Puppet freshness on tantalum is CRITICAL: Last successful Puppet run was Wed 26 Mar 2014 05:12:06 PM UTC [05:16:54] (03CR) 10Mattflaschen: "Could exposing the core file to all Gerrit users like that reveal any sensitive information in RAM?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119225 (owner: 10Faidon Liambotis) [05:19:02] (03PS1) 10Chad: All wikis with 100k pages or less can get Cirrus too [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121305 [05:22:58] (03PS3) 10Gilles: Add ?download parameter to images [operations/puppet] - 10https://gerrit.wikimedia.org/r/120617 [05:25:31] (03CR) 10Gilles: "Thanks for the tip! I was able to test this locally. I think I've taken these regular expressions as far as they can be taken. The "downlo" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120617 (owner: 10Gilles) [05:26:55] (03CR) 10Chad: [C: 032] All wikis with 100k pages or less can get Cirrus too [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121305 (owner: 10Chad) [05:27:02] (03Merged) 10jenkins-bot: All wikis with 100k pages or less can get Cirrus too [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121305 (owner: 10Chad) [05:27:24] !log demon synchronized noncirrus.dblist [05:27:50] !log demon synchronized wmf-config/InitialiseSettings.php [05:27:56] Logged the message, Master [05:30:29] I do not know if someone can look at this labs related bug https://bugzilla.wikimedia.org/show_bug.cgi?id=63152 [05:30:46] it has killed many tools [05:34:26] .. right something happened .. it now does work [06:55:31] congrats ^demon|zzz [07:17:40] PROBLEM - Disk space on db1044 is CRITICAL: DISK CRITICAL - free space: / 143 MB (2% inode=69%): [07:32:40] RECOVERY - Disk space on db1044 is OK: DISK OK [07:56:09] (03CR) 10Tobias Gritschacher: "I guess we do not have to wait until the banner is fixed. The current situation where people can get to en.wikidata.org and assume everyth" [operations/dns] - 10https://gerrit.wikimedia.org/r/119032 (owner: 10Faidon Liambotis) [08:09:34] (03PS10) 10Ori.livneh: Change home directory of vagrant user [operations/puppet] - 10https://gerrit.wikimedia.org/r/118053 (owner: 10Physikerwelt) [08:09:39] (03CR) 10Ori.livneh: [C: 032] Change home directory of vagrant user [operations/puppet] - 10https://gerrit.wikimedia.org/r/118053 (owner: 10Physikerwelt) [08:16:20] PROBLEM - Puppet freshness on tantalum is CRITICAL: Last successful Puppet run was Wed 26 Mar 2014 05:12:06 PM UTC [08:40:08] !log springle synchronized wmf-config/db-eqiad.php 's7 pool db1034, warm up' [08:40:14] Logged the message, Master [09:56:35] (03PS1) 10Springle: s7 pool db1034 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121337 [09:56:54] (03CR) 10Springle: [C: 032] s7 pool db1034 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121337 (owner: 10Springle) [09:57:01] (03Merged) 10jenkins-bot: s7 pool db1034 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121337 (owner: 10Springle) [09:57:37] !log springle synchronized wmf-config/db-eqiad.php 's7 db1034 full steam' [09:57:43] Logged the message, Master [10:02:37] !log db1034 running custom mariadb 5.5.34 build. if in doubt, depool [10:02:42] Logged the message, Master [10:02:57] (03PS2) 10ArielGlenn: download.wikimedia.org to misc web cluster with dataset1001 backend [operations/puppet] - 10https://gerrit.wikimedia.org/r/120998 [10:07:40] hi springle [10:08:04] aude: hi ;) [10:08:20] wondering if we ever did the schema changes for wb_terms table? [10:08:28] 1) https://gerrit.wikimedia.org/r/#/c/99637/ make row ids use bigint [10:08:39] 2) https://gerrit.wikimedia.org/r/#/c/99660/ composite indexes [10:09:03] * springle looks [10:09:27] thanks [10:10:05] there must be bugzilla ticket for those [10:10:33] https://bugzilla.wikimedia.org/show_bug.cgi?id=60540 [10:10:44] https://bugzilla.wikimedia.org/show_bug.cgi?id=60539 [10:12:05] aude: the indexes have been applied. the conversion od id to bigint has not yet been applied [10:12:10] ok [10:12:31] (03PS3) 10ArielGlenn: download.wikimedia.org to misc web cluster with dataset1001 backend [operations/puppet] - 10https://gerrit.wikimedia.org/r/120998 [10:12:37] i think we were going to do something a bit different for wikidata [10:12:39] I should have commented on 99660. will do so [10:13:16] or that was for the indexes maybe [10:13:23] different how? [10:14:17] it was partitioning for indexes [10:15:08] (03CR) 10ArielGlenn: [C: 032] download.wikimedia.org to misc web cluster with dataset1001 backend [operations/puppet] - 10https://gerrit.wikimedia.org/r/120998 (owner: 10ArielGlenn) [10:18:24] aude: wb_terms is partitioned on production slaves, but iirc daniel decided not to out that into the schema, and just leave it as a WMF production optimization [10:18:32] s/out/put/ [10:18:48] that's what i remember [10:18:58] which i think was a good idea. so far partitioning is helping, but as data set grows... who knows [10:19:06] agree [10:19:18] when do you think we can do the other change? [10:19:57] not a hurry, perhaps but don't want to forget [10:20:06] i have it as low priority, but not forgotten [10:20:23] ok, that's fine [10:20:37] because wb_terms id is still only 300m, and smaller primary key means smaller secondary indexes [10:20:43] yep [10:21:35] when wikidata gets it's own shard, then we can probably pull it all into line. the way wikidatawiki grows, i've been thinking that will be sooner rather than later [10:21:43] ok [10:23:20] hi :)) [10:25:48] springle: commented on the bugs [10:26:44] aude: thank you [10:54:33] (03PS1) 10ArielGlenn: separate stanzas for http/https redirects for download.wm.o [operations/puppet] - 10https://gerrit.wikimedia.org/r/121344 [10:56:23] (03CR) 10ArielGlenn: [C: 032] separate stanzas for http/https redirects for download.wm.o [operations/puppet] - 10https://gerrit.wikimedia.org/r/121344 (owner: 10ArielGlenn) [10:59:14] (03PS1) 10ArielGlenn: missed two vars in the download.wm.o redirect fixup [operations/puppet] - 10https://gerrit.wikimedia.org/r/121349 [11:00:52] (03CR) 10ArielGlenn: [C: 032] missed two vars in the download.wm.o redirect fixup [operations/puppet] - 10https://gerrit.wikimedia.org/r/121349 (owner: 10ArielGlenn) [11:17:20] PROBLEM - Puppet freshness on tantalum is CRITICAL: Last successful Puppet run was Wed 26 Mar 2014 05:12:06 PM UTC [11:20:20] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [11:22:06] (03PS1) 10Hashar: beta: point autoupdater to use /data/project [operations/puppet] - 10https://gerrit.wikimedia.org/r/121360 [11:22:22] (03CR) 10ArielGlenn: [C: 032] dumps.wikimedia.org moved to dataset1001 [operations/dns] - 10https://gerrit.wikimedia.org/r/121000 (owner: 10ArielGlenn) [11:22:26] apergos: while you are around would you mind merging in a path change for beta please? https://gerrit.wikimedia.org/r/121360 :-] [11:22:48] !log springle synchronized wmf-config/db-eqiad.php 's5 depool db1021 for schema changes' [11:22:50] cause some file not found errors on the eqiad beta cluster because it lacks /home/wikipedia :] [11:22:53] Logged the message, Master [11:24:03] I can't right now hashar, I screwed up the https redirection on the backend of the dataset hosts and need to figure out how to fix [11:24:11] :( [11:24:29] apergos: just need +2 / merge in repo, doesn't have to be on the prod puppetmaster :/ [11:24:44] though I can cherry pick it on our puppetmaster [11:24:47] will just do that :] [11:26:05] I keep forgetting we have our own puppet in eqiad [11:27:00] hashar: sorry for not starting the backport of puppet-lint, busy as crazy [11:27:11] matanya: that is not that much of a priority :] [11:27:17] migrated production servers all week long [11:27:19] matanya: and we can still use gem if it happens to be urgent hehe [11:27:52] ugh, I didn't want this to rely on parent that won't be sommmitted, and now it's submitted and merge pending [11:27:54] sigh [11:28:25] ok, noted hashar thanks [11:28:27] (03CR) 10Hashar: [C: 031 V: 032] "cherry picked on deployment-salt.eqiad.wmflabs puppet master. That fixed the path issue. Compare:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/121360 (owner: 10Hashar) [11:28:42] apergos: you can use the cherry-pick button in Gerrit [11:28:48] apergos: and cherry pick your change to the production branch [11:28:59] this way its parent will be the tip of the branch and you can merge it :] [11:29:29] I git commmit ammend and push it after the cherry pick? [11:29:39] that's way too many mms in there [11:31:25] (03PS2) 10ArielGlenn: dumps.wikimedia.org moved to dataset1001 [operations/dns] - 10https://gerrit.wikimedia.org/r/121000 [11:32:15] (03PS3) 10ArielGlenn: dumps.wikimedia.org moved to dataset1001 [operations/dns] - 10https://gerrit.wikimedia.org/r/121000 [11:32:50] (03CR) 10ArielGlenn: [C: 032] dumps.wikimedia.org moved to dataset1001 [operations/dns] - 10https://gerrit.wikimedia.org/r/121000 (owner: 10ArielGlenn) [11:51:44] !log springle synchronized wmf-config/db-eqiad.php 's5 repool db1021 warm up' [11:51:50] Logged the message, Master [12:40:20] (03PS1) 10Hashar: beta: bring in scap related scripts on bastion [operations/puppet] - 10https://gerrit.wikimedia.org/r/121365 [12:43:33] (03CR) 10Hashar: [C: 031 V: 032] "Applied on eqiad beta cluster puppet master deployment-salt.eqiad.wmflabs. Fixed the beta-code-update-eqiad jenkins job which takes care " [operations/puppet] - 10https://gerrit.wikimedia.org/r/121365 (owner: 10Hashar) [12:56:45] paravoid: i know it isn't the place, but debian wiki is down [13:07:04] !log springle synchronized wmf-config/db-eqiad.php 's5 db1021 full steam' [13:07:10] Logged the message, Master [13:18:12] (03CR) 10ArielGlenn: "do not merge until further notice; on hold" [operations/dns] - 10https://gerrit.wikimedia.org/r/120999 (owner: 10ArielGlenn) [13:25:10] RECOVERY - Puppet freshness on tantalum is OK: puppet ran at Thu Mar 27 13:25:02 UTC 2014 [13:55:20] springle: I assume db1034 is you, correct? [13:55:31] paravoid: yeah, checking it now [13:55:37] okay, great [13:58:13] hashar: i tested the upstream debian package from jessie [13:58:27] it works ok in presice [13:58:38] you can take that, if you want [13:58:53] springle: btw, any reason you're doing "silence notifications" instead of "acknowledge" in the icinga web intf? [13:58:56] matanya: for what? [13:59:02] puppet-lint [13:59:30] [1830901.261363] mysqld[16175]: segfault at 0 ip 00000000006840d6 sp 00007f3d12452c00 error 4 in mysqld[400000+b54000] [13:59:36] that's from db1034 [14:00:42] paravoid: i forgot to renable the silenced stuff from earlier [14:00:45] and now tested the one from trusty. works ok in precise too. same dep's [14:01:14] so basicly, if ops wish, backporting is quite easy [14:01:16] springle: if you do acknowledge, it sticks until it recovers, then warns again [14:01:38] Table './heartbeat/heartbeat' is marked as crashed and should be repaired [14:01:45] fun stuff [14:04:33] !log springle synchronized wmf-config/db-eqiad.php 's7 depool db1034 after crash' [14:04:39] Logged the message, Master [14:05:07] matanya: awesome. If you could handle the communication with ops to get the puppet-lint package updated in apt.wikimedia.org that would be nice :] [14:05:20] matanya: I think I previously filled a RT then poked folks here. [14:06:51] springle: Megan (who saw server errors on www and meta) is planning a fundraising/banner test this AM, she's asking if she should proceed or if she should hold off based on the database issue?. [14:08:49] Jeff_Green: fine to go ahead. db1034 was something of a test case. it won't go back in until i figure out what killed it [14:09:00] springle: great, thank you [14:21:46] (03PS5) 10Hashar: Create roles for syslog-ng [operations/puppet] - 10https://gerrit.wikimedia.org/r/119257 [14:21:52] (03PS6) 10Hashar: Make syslog-ng basepath a parameter [operations/puppet] - 10https://gerrit.wikimedia.org/r/119256 [14:22:45] (03PS7) 10Hashar: Make syslog-ng basepath a parameter [operations/puppet] - 10https://gerrit.wikimedia.org/r/119256 [14:23:13] (03CR) 10Hashar: Make syslog-ng basepath a parameter (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119256 (owner: 10Hashar) [14:35:57] (03PS1) 10Tim Landscheidt: Tools: Forward mail for root to the admin group [operations/puppet] - 10https://gerrit.wikimedia.org/r/121377 [14:38:51] (03PS8) 10Hashar: Make syslog-ng basepath a parameter [operations/puppet] - 10https://gerrit.wikimedia.org/r/119256 [14:40:30] (03PS6) 10Hashar: Create roles for syslog-ng [operations/puppet] - 10https://gerrit.wikimedia.org/r/119257 [14:43:38] (03CR) 10Andrew Bogott: [C: 032] Make syslog-ng basepath a parameter [operations/puppet] - 10https://gerrit.wikimedia.org/r/119256 (owner: 10Hashar) [14:43:43] \O/ [14:44:23] (03CR) 10Andrew Bogott: [C: 032] Create roles for syslog-ng [operations/puppet] - 10https://gerrit.wikimedia.org/r/119257 (owner: 10Hashar) [14:44:25] manybubbles: you there? [14:44:31] pong [14:44:40] (03CR) 10coren: [C: 04-2] "I have a more general solution WIP right now." [operations/puppet] - 10https://gerrit.wikimedia.org/r/121377 (owner: 10Tim Landscheidt) [14:44:51] hey so, i think the jvm deployment stuff is working [14:45:11] i can't fully test it in analytics cluster because of network acls there, need to get access to archiva from those nodes [14:45:22] but! can we try it for your stuff somehow? [14:45:32] hmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm [14:45:34] probably [14:45:40] i'm going to work on writing up docs today, working through it with you will help me do that too [14:45:46] we can certainly see if it drops files into the right spots [14:46:04] we'll always manually bounce the Elasticsearch nodes anyway [14:46:19] well, we can just see if we can deploy them first :) [14:47:01] ottomata: you made git-fat a dependency of deployment, so it needs to now be build for trusty as well [14:47:01] step 1: deploy the .jars you need to archiva. [14:47:02] step 2: git-fat add them to some deployment repo, enable git-fat on that deployment target [14:47:02] step 3: do a deployment [14:47:07] (speaking of git-fat) [14:47:09] done done doooone! [14:47:12] wait [14:47:17] i did not mean 'done' the word [14:47:21] i meant DUN the musical sound :p [14:47:22] ok paravoid! [14:47:41] um, ok, it is just a python script [14:47:52] should I just change the distribution and add it to apt? [14:49:53] you can even reprepro copy it [14:49:59] if it works [14:50:54] oh it should, lemme see [14:55:22] paravoid, do we have wikimedia-trusty dist? [14:55:25] yes [14:55:26] is that what I shoudl copy it to? [14:55:33] yes [14:55:35] k [14:57:17] (03PS1) 10Hashar: beta: fix wrong include in role::beta::natfix [operations/puppet] - 10https://gerrit.wikimedia.org/r/121378 [14:59:09] hmm, maybe reprepro copy won't work? [14:59:12] Will not copy as not found: git-fat. [14:59:57] paravoid: you seen that before? ^ [15:00:12] the copy arguments are very counter-intuitive [15:00:14] they're opposite [15:00:17] did you notice that? [15:00:30] it's copy [15:00:36] if I recall correctly [15:00:37] i did not notice that [15:00:53] ah ha! there it goes [15:01:05] that worked, thanks! [15:01:15] (03PS1) 10Hashar: Add bastion-eqiad.wmflabs.org is a bastion_hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/121379 [15:03:02] it's like strcpy ;) [15:04:10] welllll woot! git-deploy + archiva + git-fat just worked for the first time [15:04:11] yeehaw [15:10:02] (03PS2) 10Hashar: Add labs eqiad bastions in networks.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/121379 [15:10:40] (03CR) 10Andrew Bogott: [C: 032] Add labs eqiad bastions in networks.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/121379 (owner: 10Hashar) [15:11:16] (03PS1) 10coren: Tool Labs: fixes mail to root (and postmaster) [operations/puppet] - 10https://gerrit.wikimedia.org/r/121380 [15:11:34] is wikimedia going to switch to WebScaleSQL? [15:13:09] (03CR) 10Andrew Bogott: [C: 032] beta: fix wrong include in role::beta::natfix [operations/puppet] - 10https://gerrit.wikimedia.org/r/121378 (owner: 10Hashar) [15:14:06] domas: do you still need access to stat1? [15:14:08] (03CR) 10coren: [C: 032] "Tested already." [operations/puppet] - 10https://gerrit.wikimedia.org/r/121380 (owner: 10coren) [15:14:24] matanya: what does it do, what is all that thing about revoking access, maybe I need it! [15:14:38] I don't know when I need access to things, but when I need them it is better to have it [15:14:39] :) [15:14:58] domas: ottomata is auditing access to this machine [15:15:04] i'm trying to help him [15:15:12] what is so special about that machine to audit access to it [15:15:24] it got some super private data stuffs [15:15:25] do people have nothing better to do? [15:15:28] naw it doesn't [15:15:32] domas: yup, exactly [15:15:32] :) [15:15:36] then it's boring :P [15:15:36] fine [15:15:38] then leave my access [15:15:40] :)) [15:15:47] we are migrating to a new machine [15:15:54] just not migrating things we don't need to, to clean up [15:15:55] it is being decommed, and we don't want to create them on the new replace box [15:16:01] (03PS6) 10Hoo man: Introduce an admins::release user group [operations/puppet] - 10https://gerrit.wikimedia.org/r/116019 [15:16:17] domas, you would only need access on stat1 if you intend to query research slave dbs and save the data and do computation on it [15:16:21] mh... rush doesn't have IRC, does he? [15:16:33] is does hoo [15:16:51] right, then I need access!!!! [15:16:52] his nick is chasemp [15:17:01] though I enjoy building my own stats pipelines! [15:17:15] * hoo eyes chasemp for review https://gerrit.wikimedia.org/r/116019 [15:17:31] domas: you have enough private data at facebook :P [15:17:37] domas is a root [15:17:43] as such he gets to keep access to all machines [15:17:58] yay [15:18:43] thank you mark [15:18:55] (03CR) 10Hoo man: [C: 031] "anyone?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120730 (owner: 10Hoo man) [15:20:05] lets see who is left [15:20:26] (03Abandoned) 10Hoo man: Remove deprecated wgCopyrightIcon in favor of wgFooterIcons [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110668 (owner: 10Hoo man) [15:32:02] (03PS1) 10Springle: Host was depooled from tin. Make it stick until further post-crash testing can be done. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121387 [15:40:24] (03PS1) 10Ottomata: Moving inclusion of standard and admins::roots out of analytics roles [operations/puppet] - 10https://gerrit.wikimedia.org/r/121389 [15:42:04] (03PS2) 10Ottomata: Moving inclusion of standard and admins::roots out of analytics roles [operations/puppet] - 10https://gerrit.wikimedia.org/r/121389 [15:42:22] RobH: daily question : what do i need to do in order to get a newer version to apt.wikimedia.org ? [15:42:46] matanya: , someone has to build and/or add the package there [15:42:53] newer version of what? [15:42:57] pupet-lint [15:43:01] *puppet [15:43:24] what ottomata said, as for how to do that.... [15:43:41] i guess an RT ticket detailing why and such [15:43:46] aye [15:43:59] but usually has to be a pretty detailed reason to not stick with the ubuntu defaults (cuz maintaining is a pain) [15:44:26] but i have not been involved in that for awhile so if someone else knows more correct me (1) [15:44:30] (!) even. [15:44:58] doesn't hurt that matanya can point at a long list of things demonstrating use of lint.... [15:45:04] heh [15:45:28] (03CR) 10Springle: [C: 032 V: 032] Host was depooled from tin. Make it stick until further post-crash testing can be done. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121387 (owner: 10Springle) [15:45:54] (03CR) 10Ottomata: [C: 032 V: 032] Moving inclusion of standard and admins::roots out of analytics roles [operations/puppet] - 10https://gerrit.wikimedia.org/r/121389 (owner: 10Ottomata) [15:45:57] RobH: i have backported the one from trusty, and tested the one from jessie [15:46:15] they both act nicely and sane, and have the same dependencies [15:46:32] Coren: i just merged a tool labs commit from you, hope that's ok [15:46:35] on palladium [15:47:03] so it is not much of a pain involved. i also tested on a trusty instance, worked good as well [15:47:58] so apart from pushing the debian directory (to where? operations/debs/puppet-lint?) what should i do ? [15:49:21] RobH: we can wait for trusty, i guess, but i'm not sure we need to [15:49:23] (03PS2) 10BBlack: lvs300[1234] dns for esams private subnet [operations/dns] - 10https://gerrit.wikimedia.org/r/121235 [15:49:25] so, RT ticket would be good, but if you want to make it super easy, make operations/debs/puppet-lint a git-buildpackage usable repo [15:49:31] that would mean: [15:49:35] master is just a fork of upstream [15:49:38] ottomata: It was. Sorry, I got distracted mid-process. [15:49:49] debian branch is a branch of master with changes only made to debian/ directory [15:49:59] matanya: oh not saying its a no, ottomata has more info on this than me obviously =] [15:50:01] debian/gbp.conf specifies which branches to use [15:50:06] etc. [15:50:10] then someone can just clone your repo [15:50:11] (03CR) 10BBlack: [C: 032 V: 032] lvs300[1234] dns for esams private subnet [operations/dns] - 10https://gerrit.wikimedia.org/r/121235 (owner: 10BBlack) [15:50:13] git-buildpackage [15:50:15] and have a .deb [15:50:58] so just clone the upstream source deb tree to out tree basiclly ottomata ? [15:51:35] (03PS2) 10BBlack: private1-esams dhcp/preseed stuff [operations/puppet] - 10https://gerrit.wikimedia.org/r/121236 [15:52:16] (03CR) 10BBlack: [C: 032 V: 032] private1-esams dhcp/preseed stuff [operations/puppet] - 10https://gerrit.wikimedia.org/r/121236 (owner: 10BBlack) [15:54:48] matanya, i guess it depends on what upstream's layout looks like [15:55:06] are they using a git repo? do they already have a debian directory? is it from the source package? is it from a source tarball? [15:55:20] matanya: http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.html [15:55:30] ottomata: http://packages.ubuntu.com/source/trusty/puppet-lint [15:55:33] (03PS1) 10Ottomata: role::analytics must be included before standard so that ganglia_cluster is set properly [operations/puppet] - 10https://gerrit.wikimedia.org/r/121395 [15:55:39] ahhh from ubuntu hmmm right [15:55:46] (03CR) 10jenkins-bot: [V: 04-1] role::analytics must be included before standard so that ganglia_cluster is set properly [operations/puppet] - 10https://gerrit.wikimedia.org/r/121395 (owner: 10Ottomata) [15:55:55] oh wait, there is already a deb built? [15:55:59] is this the version you want? [15:56:00] yes [15:56:08] oh its just from future? [15:56:13] yes [15:56:16] trusty [15:56:17] oh oh oh [15:56:31] works well on precise [15:56:33] i take all that back then, I don't think we need a repo…i am not certain, but we can probably just import it into apt…paravoid [15:56:34] ? [15:56:58] same deps and all that [15:57:24] (03PS2) 10Ottomata: role::analytics must be included before standard so that ganglia_cluster is set properly [operations/puppet] - 10https://gerrit.wikimedia.org/r/121395 [15:58:07] (03CR) 10Ottomata: [C: 032 V: 032] role::analytics must be included before standard so that ganglia_cluster is set properly [operations/puppet] - 10https://gerrit.wikimedia.org/r/121395 (owner: 10Ottomata) [16:02:22] greg-g: Sam's otherwise engaged so I'm going to start deploy prep for 1.23wmf20. [16:04:00] \O/ [16:08:21] (03PS1) 10Ottomata: Moving analytics ganglia cluster definition into site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/121396 [16:08:29] (03PS2) 10Ottomata: Moving analytics ganglia cluster definition into site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/121396 [16:08:34] (03CR) 10jenkins-bot: [V: 04-1] Moving analytics ganglia cluster definition into site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/121396 (owner: 10Ottomata) [16:08:50] (03CR) 10Ottomata: [C: 032 V: 032] Moving analytics ganglia cluster definition into site.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/121396 (owner: 10Ottomata) [16:12:08] (03CR) 10CSteipp: "It would if there was anything sensitive in the process or environment, which is why all hardening guides tell you to turn off core dumps." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119225 (owner: 10Faidon Liambotis) [16:13:27] akosiaris: Did I hear that you have some magic way to detect whether or not a puppet patch is a no-op? https://gerrit.wikimedia.org/r/#/c/120005/2 [16:15:01] andrewbogott: he has, it is called catalog-differ [16:15:28] bd808|deploy: alrighty then [16:15:39] (03Abandoned) 10Tim Landscheidt: Tools: Forward mail for root to the admin group [operations/puppet] - 10https://gerrit.wikimedia.org/r/121377 (owner: 10Tim Landscheidt) [16:15:45] * bd808|deploy is cutting the branch now [16:15:54] and i'm waiting to when he releases it with oss lisence [16:16:38] ottomata: i'f you hear from paravoid on the package thingy, please let me know. i'm out, see you. [16:20:41] (03PS1) 10Mglaser: Added Markus Glaser's GPG keys [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121397 [16:21:14] (03PS1) 10Tim Landscheidt: Tools: Work around problem with mail for nested service groups [operations/puppet] - 10https://gerrit.wikimedia.org/r/121398 [16:21:42] (03CR) 10Hashar: contint: override .jshintrc file on gallium (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119750 (owner: 10Hashar) [16:23:15] (03PS3) 10Hashar: contint: override .jshintrc file on gallium [operations/puppet] - 10https://gerrit.wikimedia.org/r/119750 [16:23:31] (03PS4) 10Hashar: contint: override .jshintrc file on gallium [operations/puppet] - 10https://gerrit.wikimedia.org/r/119750 [16:25:03] hashar: so why isn't this *in* role::ci::slave? [16:25:11] and the Mount as well [16:25:33] paravoid: we need it on the host having both the integration/docroot.git repository and the role::ci::slave. [16:25:46] paravoid: that is merely a hack for gallium :/ [16:25:50] ? [16:26:26] on gallium we have integration/docroot.git cloned at /srv/ and it provides a non default .jshintrc file as /srv/.jshintrc [16:27:00] why do we have it at /srv instead of /srv/www or something? [16:27:02] then we have role::ci::slaves which setup jenkins workspaces under /srv/ssd/jenkins-slave/workspace/ Whenever a jshint job run on gallium host, jshint will fallback to use the /srv/.jshintrc , causing jobs to fail [16:27:17] tech debt [16:27:37] (03CR) 10Rush: "to quote faidon "What's the rationale for this?"" [operations/puppet] - 10https://gerrit.wikimedia.org/r/116019 (owner: 10Hoo man) [16:27:49] we have the sites listed as /srv/org/wikimedia/integration/ (for http://integration.wikimedia.org/ ) [16:28:12] chasemp: You're rush, right? [16:28:18] which is what we are doing for blog / bugzilla / smoke ping etc [16:28:30] hoo: true [16:29:15] chasemp: Ok, point is to not give everyone access to private data... the users taht don't need the access should not have it [16:29:26] That's pretty obvious I hope [16:29:58] that's why I created a separate "group" for the release users so that they only get access to the bastion and their release machine [16:30:13] (03CR) 10Hashar: lvs: generic::upstart_job() now uses boolean values (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/118717 (owner: 10Hashar) [16:30:32] (03PS4) 10Hashar: lvs: generic::upstart_job() now uses boolean values [operations/puppet] - 10https://gerrit.wikimedia.org/r/118717 [16:31:12] hashar: okay, fix that then :) [16:31:30] paravoid: no way I am going to pill even more puppet changes in my dashboard to fix all the /srv/org/ paths [16:32:02] (03CR) 10Hashar: openstack: generic::upstart_job() now uses boolean values (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/118716 (owner: 10Hashar) [16:32:12] (03PS4) 10Hashar: openstack: generic::upstart_job() now uses boolean values [operations/puppet] - 10https://gerrit.wikimedia.org/r/118716 [16:32:16] no way I'm gonna merge more hackish workarounds on the ground that we have tech debt :) [16:32:40] fine to me [16:32:41] :-D [16:33:29] it just piss me of that providing a .jshintrc file containing {} requires me to spend a couple days refactoring all the jenkins jobs / apache conf relying on websites being at /srv/org/* [16:34:21] you've built all of the manifests, who are you blaming for the tech debt? [16:34:32] (03CR) 10Hashar: twemproxy: generic::upstart_job() now uses boolean values (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/118718 (owner: 10Hashar) [16:34:37] (03PS4) 10Hashar: twemproxy: generic::upstart_job() now uses boolean values [operations/puppet] - 10https://gerrit.wikimedia.org/r/118718 [16:34:46] hoo: I understand there is a creation of a sub-group, but I don't have the context knowledge to know if. a) those are the right ppl to be in it. b) if it's worth creating. the context I'm missing is the usage context. these users do x that needs y, and these users do x that needs z. let's separate them. I may not be the best reviewer but for me overall it is not clear. [16:35:40] private data is ambigious to me in this context, some is obviously more private than others [16:35:50]