[00:07:26] (03PS1) 10Ori.livneh: Add eventlogging::service::reporter for reporting to StatsD [operations/puppet] - 10https://gerrit.wikimedia.org/r/123139 [00:09:10] (03CR) 10Ori.livneh: [C: 032] Add eventlogging::service::reporter for reporting to StatsD [operations/puppet] - 10https://gerrit.wikimedia.org/r/123139 (owner: 10Ori.livneh) [00:11:24] (03PS1) 10Ori.livneh: Follow-up to I595f2c15d: add missing reporter.erb [operations/puppet] - 10https://gerrit.wikimedia.org/r/123140 [00:12:54] (03CR) 10Ori.livneh: [C: 032] Follow-up to I595f2c15d: add missing reporter.erb [operations/puppet] - 10https://gerrit.wikimedia.org/r/123140 (owner: 10Ori.livneh) [00:27:00] (03PS1) 10Ori.livneh: eventlogging: make statsd:// consumer write to tungsten [operations/puppet] - 10https://gerrit.wikimedia.org/r/123143 [00:29:00] (03CR) 10Ori.livneh: [C: 032] eventlogging: make statsd:// consumer write to tungsten [operations/puppet] - 10https://gerrit.wikimedia.org/r/123143 (owner: 10Ori.livneh) [00:30:36] PROBLEM - Kafka Broker Messages In on analytics1021 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 988.204227577 [00:43:42] (03PS1) 10Tim Landscheidt: Tools: Alias tools.wmflabs.org to internal webproxy [operations/puppet] - 10https://gerrit.wikimedia.org/r/123149 [00:51:56] PROBLEM - RAID on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:51:59] (03CR) 10Legoktm: Use the BetaFeatures whitelist for production to avoid accidental deploys (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/121892 (owner: 10Jforrester) [00:52:13] springle: db1047 again :/ [00:52:36] PROBLEM - DPKG on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:52:53] * springle sigh [00:52:56] RECOVERY - RAID on db1047 is OK: OK: optimal, 3 logical, 6 physical [00:53:26] RECOVERY - DPKG on db1047 is OK: All packages OK [01:00:54] ebernhardson: did you update the submodules before syncing? [01:01:30] ebernhardson: i don't think you did [01:01:37] i'll do the bump / sync now [01:02:30] !log ori synchronized php-1.23wmf20/extensions/EventLogging [01:02:34] Logged the message, Master [01:02:44] !log ori synchronized php-1.23wmf20/extensions/WikimediaEvents [01:02:48] Logged the message, Master [01:04:12] Someone knows about the SSL issues on gerrit transactions, right? [01:04:56] !log ori synchronized php-1.23wmf19/extensions/EventLogging [01:05:01] Logged the message, Master [01:05:15] !log ori synchronized php-1.23wmf19/extensions/WikimediaEvents [01:05:19] Logged the message, Master [01:16:21] !log ori synchronized php-1.23wmf19/extensions/WikimediaEvents 'Undeployed change from earlier SWAT deploy' [01:16:29] Logged the message, Master [01:16:31] !log ori synchronized php-1.23wmf20/extensions/WikimediaEvents 'Undeployed change from earlier SWAT deploy' [01:16:35] Logged the message, Master [01:21:06] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [01:30:14] (03PS1) 10Ori.livneh: Use utf-8 as default codec in mwgrep [operations/puppet] - 10https://gerrit.wikimedia.org/r/123156 [01:32:04] (03CR) 10Ori.livneh: [C: 032] Use utf-8 as default codec in mwgrep [operations/puppet] - 10https://gerrit.wikimedia.org/r/123156 (owner: 10Ori.livneh) [02:03:16] !log springle synchronized wmf-config/db-eqiad.php 's3 depool db1027 for upgrade' [02:03:24] Logged the message, Master [02:18:36] (03PS1) 10Springle: Remove unneeded client packages. The wmf-mariadb package provides the usual command line utilities, and other packages are free to use the usual stock libmysqlclient* dependencies without conflict. [operations/puppet] - 10https://gerrit.wikimedia.org/r/123160 [02:22:23] !log springle synchronized wmf-config/db-eqiad.php 's3 repool db1027, warm up' [02:22:27] Logged the message, Master [02:24:03] (03CR) 10Springle: [C: 032] Remove unneeded client packages. The wmf-mariadb package provides the usual command line utilities, and other packages are free to use the u [operations/puppet] - 10https://gerrit.wikimedia.org/r/123160 (owner: 10Springle) [02:28:35] (03PS1) 10coren: Tool Labs: Fix exim4 config [operations/puppet] - 10https://gerrit.wikimedia.org/r/123162 [02:29:18] !log LocalisationUpdate completed (1.23wmf19) at 2014-04-02 02:29:18+00:00 [02:29:22] Logged the message, Master [02:31:12] (03PS2) 10coren: Tool Labs: Fix exim4 config [operations/puppet] - 10https://gerrit.wikimedia.org/r/123162 [02:31:32] (03PS1) 10Springle: Add db1027 to coredb topology in S3. This omission caused db1027 to only have the standard icinga checks. [operations/puppet] - 10https://gerrit.wikimedia.org/r/123163 [02:34:27] (03CR) 10coren: [C: 032] Tool Labs: Fix exim4 config [operations/puppet] - 10https://gerrit.wikimedia.org/r/123162 (owner: 10coren) [02:35:57] (03PS2) 10Springle: Add db1027 to coredb topology in S3. This omission caused db1027 to only have the standard icinga checks. [operations/puppet] - 10https://gerrit.wikimedia.org/r/123163 [02:37:54] (03CR) 10Springle: [C: 032] Add db1027 to coredb topology in S3. This omission caused db1027 to only have the standard icinga checks. [operations/puppet] - 10https://gerrit.wikimedia.org/r/123163 (owner: 10Springle) [02:52:48] !log LocalisationUpdate completed (1.23wmf20) at 2014-04-02 02:52:48+00:00 [02:52:53] Logged the message, Master [02:55:41] !log springle synchronized wmf-config/db-eqiad.php 's2 depool db1063 for upgrade' [02:55:45] Logged the message, Master [03:04:26] !log springle synchronized wmf-config/db-eqiad.php 's2 repool db1063, warm up' [03:04:30] Logged the message, Master [03:07:55] !log springle synchronized wmf-config/db-eqiad.php 's4 depool db1059 for upgrade' [03:07:58] Logged the message, Master [03:21:36] !log springle synchronized wmf-config/db-eqiad.php 's4 repool db1059, warm up' [03:21:41] Logged the message, Master [03:27:33] !log springle synchronized wmf-config/db-eqiad.php 's5 depool db1045 for upgrade' [03:27:37] Logged the message, Master [03:36:39] (03PS1) 10coren: Labs: prevent ec2id fact from returing errors [operations/puppet] - 10https://gerrit.wikimedia.org/r/123168 [03:37:32] (03CR) 10jenkins-bot: [V: 04-1] Labs: prevent ec2id fact from returing errors [operations/puppet] - 10https://gerrit.wikimedia.org/r/123168 (owner: 10coren) [03:38:36] PROBLEM - Host db1045 is DOWN: PING CRITICAL - Packet loss = 100% [03:38:55] (03PS2) 10coren: Labs: prevent ec2id fact from returing errors [operations/puppet] - 10https://gerrit.wikimedia.org/r/123168 [03:39:40] (03PS3) 10coren: Labs: prevent ec2id fact from returing errors [operations/puppet] - 10https://gerrit.wikimedia.org/r/123168 [03:40:30] (03CR) 10jenkins-bot: [V: 04-1] Labs: prevent ec2id fact from returing errors [operations/puppet] - 10https://gerrit.wikimedia.org/r/123168 (owner: 10coren) [03:41:37] Syntax error at '}'; expected '}' [03:41:45] That is soooo helpful. really. [03:42:18] :) [03:43:48] !log springle synchronized wmf-config/db-eqiad.php 's5 repool db1045, warm up' [03:43:52] Logged the message, Master [03:43:52] Hm. The fail function documentation isn't as useful as I wanted. I was hoping it'd make the run fail when executed, not make the run fail to parse. [03:46:01] !log springle synchronized wmf-config/db-eqiad.php 's6 depool db1006 for upgrade' [03:46:03] (03PS4) 10coren: Labs: prevent ec2id fact from returing errors [operations/puppet] - 10https://gerrit.wikimedia.org/r/123168 [03:46:06] Logged the message, Master [03:46:18] Ah, or perhaps not. Maybe I used fail wrong. Fail at fail. :-) [03:46:42] which is a success, sort of :) [03:48:34] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 2 03:48:31 UTC 2014 (duration 48m 30s) [03:48:38] Logged the message, Master [03:56:43] !log springle synchronized wmf-config/db-eqiad.php 's6 repool db1006, warm up' [03:56:47] Logged the message, Master [04:27:21] !log springle synchronized wmf-config/db-eqiad.php 's7 depool db1039 for upgrade' [04:27:26] Logged the message, Master [04:42:21] !log springle synchronized wmf-config/db-eqiad.php 's7 repool db1039, warm up' [04:42:26] Logged the message, Master [04:45:09] !log springle synchronized wmf-config/db-eqiad.php 's1 depool db1062 for upgrade' [04:45:12] Logged the message, Master [04:52:20] (03CR) 10Andrew Bogott: [C: 032] "Looks good, but I'm not merging because I want to be awake when this applies everywhere..." [operations/puppet] - 10https://gerrit.wikimedia.org/r/123168 (owner: 10coren) [04:53:11] !log springle synchronized wmf-config/db-eqiad.php 's1 repool db1062, warm up' [04:53:16] Logged the message, Master [05:31:31] !log springle synchronized wmf-config/db-eqiad.php 'normal loads for all upraded slaves' [05:31:35] Logged the message, Master [05:46:06] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [06:56:43] (03PS5) 10Hashar: Labs: prevent ec2id fact from returing errors [operations/puppet] - 10https://gerrit.wikimedia.org/r/123168 (owner: 10coren) [06:57:32] (03CR) 10Hashar: "Amended topic to poke bug 63322" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/123168 (owner: 10coren) [07:03:21] <_joe_> hi all [07:59:46] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [08:00:46] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.2 (protocol 2.0) [08:08:47] (03PS3) 10Matanya: Puppetize purge-checkuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/122617 (owner: 10Hoo man) [08:13:46] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [08:15:04] (03PS1) 10Dzahn: authorize Giuseppe for all Icinga commands [operations/puppet] - 10https://gerrit.wikimedia.org/r/123179 [08:15:08] _joe_: ^ [08:16:46] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.2 (protocol 2.0) [08:18:29] (03CR) 10Giuseppe Lavagetto: [C: 031] "Username and group permissions are correct." [operations/puppet] - 10https://gerrit.wikimedia.org/r/123179 (owner: 10Dzahn) [08:20:00] (03PS2) 10Dzahn: authorize Giuseppe for all Icinga commands [operations/puppet] - 10https://gerrit.wikimedia.org/r/123179 [08:38:36] (03CR) 10Dzahn: [C: 04-1] "we need to add the matching Icinga contact first in private repo, which is pending the sms gateway address for local provider" [operations/puppet] - 10https://gerrit.wikimedia.org/r/123179 (owner: 10Dzahn) [08:52:46] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [08:53:28] (03CR) 10Dzahn: [C: 032] authorize Giuseppe for all Icinga commands [operations/puppet] - 10https://gerrit.wikimedia.org/r/123179 (owner: 10Dzahn) [08:53:40] (03PS5) 10Matanya: mail :lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/109514 [08:55:44] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.2 (protocol 2.0) [08:56:27] aha, we've got a new ops [08:57:38] MaxSem: yes, we do:) [08:57:44] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [08:57:50] _joe_: ^ that's Max, he's in mobile [08:58:01] MaxSem: it's _joe_ [08:58:13] hi _joe_ :) [08:58:49] .ru - .it [08:58:55] <_joe_> MaxSem: hi [09:06:33] so what you'll be doing? [09:06:58] (03CR) 10Dzahn: [C: 032] "shut down yesterday" [operations/dns] - 10https://gerrit.wikimedia.org/r/122837 (owner: 10Matanya) [09:08:50] i hope not a assigned to special project too soon, heh [09:09:32] !log DNS update - removing ms10 [09:09:36] Logged the message, Master [09:13:54] (03CR) 10Dzahn: [C: 032] capella: decom [operations/dns] - 10https://gerrit.wikimedia.org/r/122719 (owner: 10Matanya) [09:14:26] (03PS1) 10Giuseppe Lavagetto: Adding Giuseppe to ichinga contact groups. [operations/puppet] - 10https://gerrit.wikimedia.org/r/123186 [09:14:29] !log DNS update - removing capella [09:14:32] Logged the message, Master [09:14:54] (03CR) 10QChris: "(Keeping CR-1, to mark dependency on merge of https://gerrit.wikimedia.org/r/#/c/121531/" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/121546 (owner: 10Ottomata) [09:16:22] (03CR) 10Dzahn: [C: 032] "RT #7173" [operations/puppet] - 10https://gerrit.wikimedia.org/r/123186 (owner: 10Giuseppe Lavagetto) [09:30:21] CUSTOM - Host magnesium is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [09:30:36] _joe_: ^ works, very niceee [09:30:39] <_joe_> eheh [09:31:05] ok, that shows as you can also do the other commands [09:31:17] morning (kind of) [09:31:29] f.e. ACK when something is critical but you know it's maintenance [09:31:36] hi hashar [09:31:50] what is that? [09:32:26] paravoid: "custom notificatio" in icinga web ui to test if he is authorized for commands [09:32:48] ah [09:32:50] it doesnt do anything except send a custom notification [09:33:33] <_joe_> just to test my privileges [09:34:52] !log restarting gitblit on antimony [09:34:56] Logged the message, Master [09:35:04] that should also cause a recovery btw [09:35:26] ACKNOWLEDGEMENT - gitblit.wikimedia.org on antimony is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Temporarily Unavailable - 516 bytes in 0.013 second response time daniel_zahn restarted it [09:35:45] ACK for a critical service example [09:38:03] RECOVERY - gitblit.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 503318 bytes in 9.305 second response time [09:39:46] Coren: does virt0 still do anything [09:40:14] it runs pmtpa labs?:) [09:40:24] it is supposed to live for a bit more [09:40:34] mutante: remind you of hume :) [09:40:57] MaxSem: well yea, i just ask because there was the switch of wikitech and puppetmaster yesterday [09:41:05] it already is not a puppetmaster or webserver [09:41:16] anymore [09:41:28] matanya: ok [09:42:35] (03PS1) 10Alexandros Kosiaris: Create u_kolossos db and user [operations/puppet] - 10https://gerrit.wikimedia.org/r/123191 [09:42:44] matanya: "remove db67 from coredb,decom db64,db65,db66,db70" let's just turn that into "remove db67" and kill the others already [09:42:55] agreed [09:42:56] !log rsynced brewster /srv to carbon [09:42:57] it's just DHCP but they're off the list [09:43:00] Logged the message, Master [09:44:40] mutante: you will handle that one? did you speak to springle ? [09:45:10] matanya: yep [09:45:17] thanks [09:45:27] no, didn't speak on IRC, just gerrit comment [09:45:58] ok, your call [09:51:54] (03PS3) 10Matanya: webserver: fixing duplicate declaration of apache-mpm [operations/puppet] - 10https://gerrit.wikimedia.org/r/112423 [09:55:10] akosiaris: https://gerrit.wikimedia.org/r/#/c/116936/ ? [09:55:25] matanya: correction, i already asked on gerrit [09:55:48] mutante: you can do that ^ one too if you wish [09:56:00] "## imminent decomission/reclaim from pmtpa pending 12th floor reorg " [09:56:11] what does it mean: remove from puppet or not [09:56:55] where do you see that? [09:56:56] decom or reclaim ,pls [09:57:12] line 643 of site.pp [09:57:21] eh, or 651 in the patch [09:57:31] akosiaris: if you are around, how long does it takes your script to compile catalogs? And did you get added to some git repository somewhere? [09:58:55] mutante: you should ask springle or RobH i guess [09:59:38] (03PS2) 10Matanya: statistics: converted iptables to ferm rule [operations/puppet] - 10https://gerrit.wikimedia.org/r/117670 [09:59:39] hashar: on a 4 cpu labs machine around 30-40 minutes. To your second question the answer is "I have not yet got to it" [10:00:00] ah that takes a while :-( [10:00:22] embarrassingly parallel though [10:00:40] gimme quantum computing and I will be done in a jiffie :-) [10:01:03] you might find time/place for improvement [10:01:37] akosiaris: I am pretty sure we can find out how to make it faster whenever it is properly open sourced / published :-D [10:02:44] hashar: Yeah, I need to find some time for that. Hopefully next-next week I will be done with OSM and ops meeting and will do exactly that [10:02:56] matanya: kind of reluctant on https://gerrit.wikimedia.org/r/#/c/116936 [10:03:06] well push , announce, get folks to play with it, merge proposed enhancements :] [10:03:30] I can not find id deploy/id_rsa.pub is being used anywhere, but... [10:03:32] matanya: same here, i skipped that key back in the original patch on purpose, not just missed it:) [10:03:34] if* [10:03:50] fair enough [10:04:10] if you merge, we can find out who is using :D [10:04:15] I am wondering whether we should merge it and see what breaks [10:04:16] if any [10:04:16] exactly [10:05:04] anyway, your call [10:06:05] mutante: maybe ask bd808|BUFFER ? [10:06:24] I suppose with all the scap work, he and ori might be the best source around ? [10:07:07] i'll poke him on SF times tonight [10:07:31] akosiaris: yea, i'd rather not break it right now, just half day work today [10:07:34] and off tomorrow [10:07:57] i'd prefer to get some other decom stuff in [10:08:06] but asking them is a good idea,yep [10:09:15] today i learned: vim let you save a file encrypted with command X [10:09:36] <_joe_> hashar: encrypted how? [10:09:46] <_joe_> with gpg? [10:10:01] (03PS1) 10Hashar: ganglia: define in subclass at the top level [operations/puppet] - 10https://gerrit.wikimedia.org/r/123195 [10:10:01] it ask for some key [10:10:03] (03PS1) 10Hashar: ganglia: lint manifest! [operations/puppet] - 10https://gerrit.wikimedia.org/r/123196 [10:10:08] so a very weak encryption I guess [10:10:49] (03CR) 10Hashar: "Took me a while to find out the collector/aggregator classes since I was gripping for ganglia::collector hehe." [operations/puppet] - 10https://gerrit.wikimedia.org/r/123195 (owner: 10Hashar) [10:11:14] <_joe_> I'm sure I can find some 'easy' gpg-to-emacs integration just for the sake of an editor war, anyway :P [10:11:17] akosiaris: https://gerrit.wikimedia.org/r/#/c/123196/ could use your catalog script of doom :-] It is a huge lint [10:13:37] hashar: vim can do blowfish from 7.3 and on, so strong encryption. But <7.3 is is pkzip which is indeed weak and insecure [10:14:31] (03PS4) 10Matanya: icinga: replace iptable with ferm rules [operations/puppet] - 10https://gerrit.wikimedia.org/r/117674 [10:16:26] hashar: doing 123196... it is together with two others on the same machine so no 30 mins but considerably more :-( [10:16:43] tis ok [10:16:51] by the end of the afternoon we will get some report [10:17:35] akosiaris: want me to get your script wrapped in a jenkins job? So folks will be able to trigger the job manually by simply filling the change they are interested in [10:17:49] this way we dont have to contextswitch(self) :] [10:18:15] (03PS5) 10Alexandros Kosiaris: sockpuppet: remove leftovers [operations/puppet] - 10https://gerrit.wikimedia.org/r/116936 (owner: 10Matanya) [10:19:52] (03CR) 10Alexandros Kosiaris: "Bryan, Ori: I added you because of the scap work and you might (or might not) be knowing the answer to the following question "Is the del" [operations/puppet] - 10https://gerrit.wikimedia.org/r/116936 (owner: 10Matanya) [10:21:30] (03Abandoned) 10Hashar: Noop change (DO NOT SUBMIT) [operations/debs/ganglia] - 10https://gerrit.wikimedia.org/r/122833 (owner: 10Hashar) [10:22:44] mutante: are you using vim? There is a plugin to nicely align puppet arrows : https://github.com/godlygeek/tabular [10:23:27] hashar: yes I want it added to jenkins for sure, but let me first open source it so people can indeed improve it. Btw what is the API for jenkins ? What do you expect from the script ? exit codes/something else ? [10:23:30] hashar: yes, i use vim, that's interesting [10:24:34] (03CR) 10Hashar: [C: 031] lint role/deployment (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/122338 (owner: 10Dzahn) [10:25:07] mutante: there is a puppet git://github.com/rodjek/vim-puppet.git not sure what it does though [10:25:56] the github username sounds familiar [10:26:11] akosiaris: whatever input the script uses can be made a parameter in the Jenkins jobs for folks to manually fill. The exit code is pretty simple: exit 0 means job success, exit 1 means job error [10:26:38] akosiaris: if your shell script uses -e , any error in the script will abort the build and make it a failure [10:27:46] :-) [10:28:15] we can also have Jenkins collect some output files and attach them to the build page in the web interface. [10:28:40] for mw/core we record the debug log, test results and the LocalSettings.php that got used. [10:29:59] (03PS2) 10Hashar: ganglia: class defined in class moved to top [operations/puppet] - 10https://gerrit.wikimedia.org/r/123195 [10:49:36] hashar / mutante i use https://github.com/scrooloose/syntastic [10:50:31] matanya: ok.. now back to hume.. did you say the cron job is running? [10:50:43] tim saied it is not [10:50:51] should be on terbium [10:50:59] so hume is good to go [10:51:25] yea, i meant "running on terbium",, did it work and get merged after revert [10:51:28] looks [10:51:45] should have [10:52:18] akosiaris: can have u_aude (or maybe u_wikidata_geo) db and user for postgres? [10:52:23] do i make a bug ticket? [10:52:51] ah, can make gerrit patch [10:53:50] (03CR) 10Dzahn: [C: 04-1] "got it, thanks Sean. Diederik?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/111152 (owner: 10Diederik) [10:54:14] aude: yes, please update bz ticket though [10:54:17] ok [10:54:35] https://bugzilla.wikimedia.org/show_bug.cgi?id=63382 [10:54:36] i suppose i'll wait for kolossos patch to be merged [10:54:42] don't want merge conflicts [10:55:02] no need, update the ticket, I 'll update the patch [10:56:02] ok [10:56:23] (03CR) 10Dzahn: "mingle.corp.wikimedia.org looks broken.. is that wikimedia.mingle.thoughtworks.com now? how about redirecting to the new one?" [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/120341 (owner: 1001tonythomas) [10:56:29] done [10:57:53] (03CR) 10Dzahn: [C: 032] "because the old URL doesn't work anymore now (just a little while ago both worked)" [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/120341 (owner: 1001tonythomas) [10:58:08] (03CR) 10Matanya: ganglia: lint manifest! (034 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/123196 (owner: 10Hashar) [10:58:41] (03CR) 10Dzahn: [V: 032] "because the old URL doesn't work anymore now (just a little while ago both worked)" [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/120341 (owner: 1001tonythomas) [11:01:59] (03CR) 10Dzahn: decom: hume (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/122605 (owner: 10Matanya) [11:03:11] (03CR) 10Dzahn: "Change-Id: I448bf155c2436ac07506c7390d1d63cb620a0ebf is not merged yet, but since that doesn't actually run on hume that would not block i" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122605 (owner: 10Matanya) [11:04:23] (03PS2) 10Alexandros Kosiaris: Create u_kolossos,u_aude dbs and users [operations/puppet] - 10https://gerrit.wikimedia.org/r/123191 [11:05:01] (03CR) 10Matanya: decom: hume (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/122605 (owner: 10Matanya) [11:05:15] matanya: did anyone say if we want logs or don't want logs? [11:05:23] hoo said we don't [11:05:30] tim said he doesn't care [11:05:33] akosiaris: thanks [11:05:44] aude: you are welcome [11:05:44] matanya: fair, thanks [11:07:03] (03CR) 10Dzahn: [C: 032] Puppetize purge-checkuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/122617 (owner: 10Hoo man) [11:09:52] mark: can you please look at https://gerrit.wikimedia.org/r/#/c/118480/ ? [11:10:03] (03CR) 10Dzahn: "this has been created on terbium now. notice: /Stage[main]/Misc::Maintenance::Purge_checkuser/Cron[purge-checkuser]/ensure: created" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122617 (owner: 10Hoo man) [11:11:09] mutante: i pushed hume decom patches, if you didn't notice. letting you know, to prevent duplicate work [11:12:11] matanya: i know, we just commented both on one of them [11:12:35] * matanya facepalms [11:20:43] !log running CheckUser/maintenance/purgeOldData.php on all wikis [11:20:47] Logged the message, Master [11:21:19] (03PS1) 10Alexandros Kosiaris: brewster to carbon migration [operations/puppet] - 10https://gerrit.wikimedia.org/r/123202 [11:22:59] TimStarling: ^ running that manually once, works.. the cronjob got created on terbium as well [11:23:27] (as apache) [11:25:17] (03CR) 10Dzahn: "i ran this manually once on terbium, as apache and watching it..since this did not run for quite a while" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122617 (owner: 10Hoo man) [11:28:51] (03PS1) 10Alexandros Kosiaris: Move brewster services to carbon [operations/dns] - 10https://gerrit.wikimedia.org/r/123205 [11:28:53] (03PS1) 10Alexandros Kosiaris: Remove DNS entries for brewster [operations/dns] - 10https://gerrit.wikimedia.org/r/123206 [11:33:33] mutante: apergos wanted to keep ms10. maybe note it in the decom ticket [11:33:43] matanya: true! [11:34:52] maybe we can use it in eqiad as a tridge replacement ? [11:34:58] or mirror [11:36:22] it takes forever to run that on loginwiki [11:36:31] i'm still watching that [11:36:51] purging check user data [11:37:33] matanya: check with Ariel and RobH i suggest [11:37:59] mutante: that is becuse we do a lot of checks there [11:38:07] and i mean A LOT [11:38:18] yep. 04:34 cu_changes table on loginwiki has 1804371 rows [11:38:32] i'll ask them about ms10 [11:38:32] as opposed to something like [11:38:38] minwiki: Purging data from cu_changes...7 rows. [11:38:51] i guess two of those are mine [11:39:07] heh [11:39:37] it was expected this is slow when it runs the first time after X [11:39:42] (03CR) 10Mark Bergsma: "Some of these have been taken over by toolserver and are still using their old management ips. For convenience, let's keep them in until T" (031 comment) [operations/dns] - 10https://gerrit.wikimedia.org/r/118480 (owner: 10Matanya) [11:39:50] that's why i did it [11:40:01] so the cron should not take that long at all then [11:40:23] (03CR) 10Tim Landscheidt: "Will this solve http://apt.wikimedia.org/ returning nothing?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/123202 (owner: 10Alexandros Kosiaris) [11:47:40] matanya: Reedy: it finished (purging).. ended with zuwikt [11:48:07] so 40 minutes ~ ? [11:49:39] matanya: 27, but since it runs each day now.. should be way less [11:50:07] we should pick an eye open on the few following days [11:50:19] it doesn't log :p [11:50:41] run it again and see? :D [11:51:11] heh, yes:) [11:51:26] Yup [11:51:30] It removed 800k rows [11:51:34] 1044164 now in cu_changes [11:52:36] PROBLEM - MySQL Replication Heartbeat on db1037 is CRITICAL: CRIT replication delay 309 seconds [11:54:08] springle: ^ dewiki and wikidatawiki db's busy [11:54:44] matanya: done, 2-3 minutes [11:54:47] :) [11:54:57] nice [11:56:36] RECOVERY - MySQL Replication Heartbeat on db1037 is OK: OK replication delay -1 seconds [11:58:13] Anything else left on hume? [11:59:46] we think .. no [12:00:06] but it appears here, see this [12:00:27] https://gerrit.wikimedia.org/r/#/c/122605/ [12:00:35] (03PS2) 10Alexandros Kosiaris: brewster to carbon migration [operations/puppet] - 10https://gerrit.wikimedia.org/r/123202 [12:00:42] https://gerrit.wikimedia.org/r/#/c/122605/1/manifests/site.pp [12:01:08] does terbium replace all of those includes? it should [12:02:26] PROBLEM - Puppet freshness on carbon is CRITICAL: Last successful Puppet run was Wed 02 Apr 2014 09:02:01 AM UTC [12:03:54] (03CR) 10Dzahn: "this was merged instead now: Change-Id: I448bf155c2436ac07506c7390d1d63cb620a0ebf" [operations/puppet] - 10https://gerrit.wikimedia.org/r/74591 (owner: 10Reedy) [12:04:03] They should've been moved a while ago [12:05:40] should we claim "deploy terbium as hume replacement" is done? [12:05:49] that's separate from decom hume [12:06:06] I'd think so [12:06:24] #4839: misc::maintenance:: home dependent entries [12:06:29] hmmm,what's that [12:07:05] ah, resolved by RobH and then just reopened automatically because of a reply [12:08:27] closing with "anyone who thinks terbium lacks something that hume had, please speak up " [12:11:27] mutante: zero wiki is going to be in the cluster? (i.e. centralauth, stewards and the rest) [12:11:51] matanya: no idea [12:14:38] (03CR) 10Dzahn: [C: 04-1] "files/misc/scripts/purge-checkuser has been deleted now in Change-Id: I448bf155c2436ac07506c7390d1d63cb620a0ebf and this has a path confli" [operations/puppet] - 10https://gerrit.wikimedia.org/r/122605 (owner: 10Matanya) [12:16:43] (03PS2) 10Matanya: decom: hume [operations/puppet] - 10https://gerrit.wikimedia.org/r/122605 [12:16:52] fixed [12:16:56] matanya: feel like finding this out? [12:16:58] #7080: scap rsync slaves (mw1010 and mw1070) missing allow for 10.64.48.0/22 [12:17:06] that links to patches that have been merged [12:17:22] i _think_ it's done [12:17:38] looking [12:19:59] looks done, i'll poke bd808|BUFFER on this too [12:20:09] a bunch of questions for him today [12:20:23] tyvm [12:23:37] (03PS1) 10Dzahn: delete empty pmtpa dsh files, remove mw10 [operations/puppet] - 10https://gerrit.wikimedia.org/r/123210 [12:24:18] (03PS2) 10Dzahn: delete empty pmtpa dsh files, remove mw10 [operations/puppet] - 10https://gerrit.wikimedia.org/r/123210 [12:24:39] Reedy: ^ [12:25:03] :) [12:25:20] (03PS3) 10Dzahn: delete empty pmtpa dsh files, remove mw10,mw40,db10 [operations/puppet] - 10https://gerrit.wikimedia.org/r/123210 [12:29:55] (03CR) 10Dzahn: [C: 032] delete empty pmtpa dsh files, remove mw10,mw40,db10 [operations/puppet] - 10https://gerrit.wikimedia.org/r/123210 (owner: 10Dzahn) [12:39:07] (03PS1) 10Dzahn: remove all Tampa appservers from DHCP [operations/puppet] - 10https://gerrit.wikimedia.org/r/123211 [12:39:40] (03PS2) 10Dzahn: remove all Tampa appservers from DHCP [operations/puppet] - 10https://gerrit.wikimedia.org/r/123211 [12:40:24] matanya: there^ one more,, and now i'm out for today (actually half day which is over) cya [12:40:30] be back on Fri [12:40:57] thanks have fun on vacation [12:41:06] thanks for the help as usual, cya [12:43:32] (03PS1) 10Hashar: parsoid: systemuser is only for production [operations/puppet] - 10https://gerrit.wikimedia.org/r/123212 [12:45:53] (03CR) 10Hashar: "Applied on deployment-salt, the beta cluster puppetmaster. Solves:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/123212 (owner: 10Hashar) [12:46:20] (03CR) 10Hashar: [C: 031 V: 032] parsoid: systemuser is only for production [operations/puppet] - 10https://gerrit.wikimedia.org/r/123212 (owner: 10Hashar) [12:50:38] (03PS1) 10MaxSem: Clean up DSH groups [operations/puppet] - 10https://gerrit.wikimedia.org/r/123213 [12:50:52] mutante|away, you might also be interested in^^^^ [12:57:44] join jenkins [13:28:36] (03CR) 10Andrew Bogott: [C: 032] Labs: prevent ec2id fact from returing errors [operations/puppet] - 10https://gerrit.wikimedia.org/r/123168 (owner: 10coren) [13:36:59] (03PS1) 10Andrew Bogott: Don't try to reset $certname [operations/puppet] - 10https://gerrit.wikimedia.org/r/123221 [13:37:10] Coren: fyi, ^ [13:39:36] (03CR) 10Andrew Bogott: [C: 032] Don't try to reset $certname [operations/puppet] - 10https://gerrit.wikimedia.org/r/123221 (owner: 10Andrew Bogott) [13:45:29] (03CR) 10Hashar: ganglia: lint manifest! (034 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/123196 (owner: 10Hashar) [13:45:44] (03PS2) 10Hashar: ganglia: lint manifest! [operations/puppet] - 10https://gerrit.wikimedia.org/r/123196 [13:48:32] andrewbogott: Ay, yes. I keep forgetting that puppet syntax only /looks/ like it makes sense. :-) [13:48:51] yep, me too :( [13:50:02] hii Reedy, you there? [13:52:01] andrewbogott: So at least that's going to prevent that same error from occuring again. As I've said, about half the tools instance got mangled puppet.conf out of it; I expect that's true of all projects and we should probably inventory. [13:54:20] ottomata: mostly [13:54:36] hey, just wondering about your stat1` account [13:54:43] you are one of the last 3 I have yet to hear from :) [13:54:46] heh [13:54:52] do you want an account on the new stat1003 replacement server? [13:54:53] I suspect I probably don't need it... [13:55:04] I can't remember the last time I logged onto it [13:55:09] you'd only need it if you need to connect to mysql research slaves AND you want a place to save query results and do number crunching [13:55:10] I think it was poking at api request logs [13:55:21] aye, cool, those aren't even on stat1/stat1003 anymore :) [13:55:33] heh [13:55:42] those are on stat1002 [13:55:45] anyway, ok cool, thanks [13:55:46] I guess that's a no then [13:55:53] andrewbogott: Could you please use the syntax "Bug: 63322" on a single line immediately prior to "Change-Id"? That triggers the Bugzilla comments by Gerrit Notification Bot. [13:56:40] heya apergos, just noticed something with users on stat1002 [13:56:52] scfc_de: ok [13:56:59] you did a couple of RTs for access there back in december [13:57:00] hashar: answered you on the patch, from some reason it doesn't seem to show up here [13:57:08] and you added the users in site.pp under the node [13:57:19] that's fine, but there is an accounts::privatedata class in admins.pp [13:57:26] Coren: thoughts on how to inventory? [13:57:26] probably did, when I didn't know about the class [13:57:27] that I've been using for users with access to private data [13:57:32] ahh, cool, you know now [13:57:35] yep [13:57:39] ok cool, just letting you know that it exists for the future [13:57:42] yep [13:57:44] cool, ok thanks! [13:58:42]