[00:13:41] RobH: on that note, what is an acceptable 'deliverable' (if you will) in terms of the work required to provision a new instance? the way initial provisioning plays out on the app servers is quite brittle so i was thinking of working on that a little [00:14:02] having a shell script pull files is fair game? [00:14:32] is there an example of a server class that you especially like, that i could turn to for inspiration? :) [00:24:11] hrmm [00:24:21] example of a cluster service i like how it deploys.... [00:24:35] thats tough, i really dont touch db class deployes beyond the OS and thats it [00:24:46] only do installs regularly on mw and cp servers [00:24:55] (and misc services but those are each different) [00:25:17] and i havent dont a new cp server install in a very long time [00:25:24] we dont add new ones very often [00:28:29] the mw servers are simply the largest pool, hence we do them the most [00:29:32] and mw servers have to use the mw deployment systems, so its a distinct difference from other clusters [00:30:46] though imo ideally shell scripts dont copy things for puppet. puppet puts files in place for non-dev-deployment fired items, and the dev-deployment handles its files. though i suppose that means that manually firing of whatever dev-deploy stuff when we resurrect servers =P [00:31:02] right now its supposed to fire shell scripts via puppet, so we do what we dont love but meh [00:31:26] ori: so ive failed to answer your question entirely ;] [00:32:52] there's no puppet<-->scap or puppet<-->git-deploy bridge atm tho [00:33:04] i mean, there sort of is but it's hacky as hell [00:33:18] didnt mw servers used to fire the sync scripts when puppet ran? [00:33:19] yea [00:33:21] its hacky as hell [00:34:07] The mw servers will run sync-common if apache is down when puppet runs but it's a little sketchy [00:34:35] yea, cool, its not changed from what i knew [00:34:44] well, not cool i guess cuz its not exactly reliable [00:34:48] but yea ;] [00:35:26] There is something similar for git-deploy as well. [00:36:50] hopefully when it migrates to that its more reliable =] [01:18:48] (03PS2) 10Springle: Update coredb topology array which is out of sync after slave moves. Remove role::coredb::m2 as it is no longer in use. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142199 [01:20:24] (03CR) 10Springle: [C: 032] Update coredb topology array which is out of sync after slave moves. Remove role::coredb::m2 as it is no longer in use. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142199 (owner: 10Springle) [01:29:01] (03Abandoned) 10Scottlee: Fixed spacing and puppet-lint issues. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142417 (owner: 10Scottlee) [01:45:00] (03PS1) 10Gergő Tisza: Keep thumbnail guessing enabled [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142476 (https://bugzilla.wikimedia.org/64554) [02:12:13] (03Abandoned) 10Jforrester: Enable VisualEditor by default on outreachwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/134849 (owner: 10Jforrester) [02:32:50] (03PS2) 10Springle: Support monitoring single and multi-source replication configurations. [operations/puppet/mariadb] - 10https://gerrit.wikimedia.org/r/142183 [02:34:32] (03PS3) 10Springle: Support monitoring single and multi-source replication configurations. [operations/puppet/mariadb] - 10https://gerrit.wikimedia.org/r/142183 [02:35:56] (03CR) 10Springle: [C: 032] Support monitoring single and multi-source replication configurations. [operations/puppet/mariadb] - 10https://gerrit.wikimedia.org/r/142183 (owner: 10Springle) [02:36:23] !log LocalisationUpdate completed (1.24wmf10) at 2014-06-27 02:35:20+00:00 [02:36:29] Logged the message, Master [02:40:47] (03PS1) 10Springle: Update mariadb submodule sha1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/142477 [02:41:52] (03PS2) 10Springle: Update mariadb submodule sha1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/142477 [02:42:26] (03CR) 10Springle: [C: 032] Update mariadb submodule sha1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/142477 (owner: 10Springle) [02:45:49] (03CR) 10Springle: master/slave configuration for m3/phabricator (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/142188 (owner: 10Springle) [02:45:56] (03PS2) 10Springle: master/slave configuration for m3/phabricator [operations/puppet] - 10https://gerrit.wikimedia.org/r/142188 [03:06:34] !log LocalisationUpdate completed (1.24wmf11) at 2014-06-27 03:05:31+00:00 [03:06:42] Logged the message, Master [03:10:18] (03PS3) 10Springle: master/slave configuration for m3/phabricator [operations/puppet] - 10https://gerrit.wikimedia.org/r/142188 [03:12:21] (03CR) 10Springle: [C: 032] master/slave configuration for m3/phabricator [operations/puppet] - 10https://gerrit.wikimedia.org/r/142188 (owner: 10Springle) [03:21:37] (03PS1) 10Springle: Missing module name. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142478 [03:23:11] (03CR) 10Springle: [C: 032] Missing module name. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142478 (owner: 10Springle) [03:31:06] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Jun 27 03:30:00 UTC 2014 (duration 29m 59s) [03:31:10] Logged the message, Master [03:32:05] (03PS1) 10Scottlee: Fixed spacing and lint rules for manifests files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142479 [03:33:31] (03CR) 10jenkins-bot: [V: 04-1] Fixed spacing and lint rules for manifests files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142479 (owner: 10Scottlee) [03:37:35] (03PS1) 10Ori.livneh: add apache::mod::proxy_fcgi [operations/puppet] - 10https://gerrit.wikimedia.org/r/142480 [03:54:49] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 20.00% of data exceeded the critical threshold [500.0] [03:59:58] (03PS1) 10Scottlee: Fixed spacing and lint rules for manifests/misc files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142483 [04:02:01] (03CR) 10jenkins-bot: [V: 04-1] Fixed spacing and lint rules for manifests/misc files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142483 (owner: 10Scottlee) [04:06:23] PROBLEM - Disk space on ms-be3003 is CRITICAL: DISK CRITICAL - free space: / 4 MB (0% inode=94%): [04:12:44] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% data above the threshold [250.0] [04:41:39] (03CR) 10Ori.livneh: [C: 032] add apache::mod::proxy_fcgi [operations/puppet] - 10https://gerrit.wikimedia.org/r/142480 (owner: 10Ori.livneh) [04:46:29] (03PS1) 10Springle: Additional replication state checks. [operations/software] - 10https://gerrit.wikimedia.org/r/142484 [04:47:12] (03CR) 10Springle: [C: 032] Additional replication state checks. [operations/software] - 10https://gerrit.wikimedia.org/r/142484 (owner: 10Springle) [04:50:54] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: Fetching origin [05:39:55] PROBLEM - Disk space on labstore1001 is CRITICAL: DISK CRITICAL - free space: /exp/dumps 379727 MB (3% inode=99%): [05:43:45] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 525 bytes in 0.002 second response time [05:45:45] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.014 second response time [06:17:34] (03PS1) 10Springle: Deploy db1072 and db1073 as future S1. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142487 [06:19:22] PROBLEM - ElasticSearch health check on elastic1003 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 19: number_of_data_nodes: 19: active_primary_shards: 2181: active_shards: 6542: relocating_shards: 0: initializing_shards: 1: unassigned_shards: 1 [06:20:17] RECOVERY - ElasticSearch health check on elastic1003 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 19: number_of_data_nodes: 19: active_primary_shards: 2182: active_shards: 6545: relocating_shards: 0: initializing_shards: 0: unassigned_shards: 0 [07:36:46] (03PS1) 10Yuvipanda: graphite: Remove auth requirement for labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/142494 [07:48:19] (03CR) 10Yuvipanda: "Tested to work at charcoal-test.wmflabs.org" [operations/puppet] - 10https://gerrit.wikimedia.org/r/142494 (owner: 10Yuvipanda) [07:52:37] matanya: ^ removes auth :) [08:06:11] ah, the silent period on ops [08:06:21] where europe hasn't fully turned up and the US is asleep [08:35:49] (03CR) 10Matanya: [C: 031] graphite: Remove auth requirement for labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/142494 (owner: 10Yuvipanda) [08:57:05] matanya: some puppet questions (re: include semantics) [08:57:55] (03PS1) 10Yuvipanda: nginx: Make nginx package name configurable [operations/puppet/nginx] - 10https://gerrit.wikimedia.org/r/142499 (https://bugzilla.wikimedia.org/67188) [08:58:33] (03PS2) 10Yuvipanda: nginx: Make nginx package name configurable [operations/puppet/nginx] - 10https://gerrit.wikimedia.org/r/142499 (https://bugzilla.wikimedia.org/67188) [08:59:08] (03PS1) 10Yuvipanda: Add .gitreview [operations/puppet/nginx] - 10https://gerrit.wikimedia.org/r/142500 [09:01:41] hmm, that won't work [09:15:06] yes YuviPanda? [09:15:32] matanya: see bug report linked above in the patch [09:15:44] matanya: I want to make dynamicproxy use the nginx module, but use a different package (nginx-extras vs -full) [09:16:08] matanya: and nginx::site includes nginx module without any params. so how do I configure the params passed to the nginx module? [09:16:46] not sure, and short on time, I'm sorry. Will need to look in the code [09:16:56] ori is your best guess here, i think [09:17:19] matanya: will do when he's up. thanks :) [09:17:41] I might be able to dig this in about 36h [09:18:51] matanya: alright. I'll poke again if it isn't resolved by then [09:19:03] sure, have a great day [09:19:45] (03CR) 10Yuvipanda: [C: 04-1] "I see my rebase has made it so that there are two packages being installed (-full and a debug), unsure what's the best way to proceed." [operations/puppet/nginx] - 10https://gerrit.wikimedia.org/r/142499 (https://bugzilla.wikimedia.org/67188) (owner: 10Yuvipanda) [10:27:44] (03PS3) 10Krinkle: Set wgGitInfoCacheDirectory to point to scap managed location [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142320 (https://bugzilla.wikimedia.org/53972) (owner: 10BryanDavis) [10:28:39] (03CR) 10Krinkle: "Fixed broken footer (empty lines terminate the body later instead of it being in the footer, just think of headers such as in Git commits," [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142320 (https://bugzilla.wikimedia.org/53972) (owner: 10BryanDavis) [11:30:49] holaaa, anyone available to help trouble shoot statsd/graphite [11:47:28] yes nuria? [11:50:47] hola matanya, we have an issue with graphite counters for EL that stopped working on may 16th [11:51:04] nuria: production graphite ? [11:51:09] yes [11:51:20] let me send you an exmaple [11:51:23] ah, sorry. can't help with that. no access [11:51:42] ok, who would be best to help with that? [11:53:26] I guess ori is your best shot nuria [11:54:04] right but our schedules are like opposite though...I will try to reach him [11:54:18] anyone besides ori that could help? [11:56:06] nuria: chasemp would also be able to help [11:58:06] ok i see , will ask chasemp later, seems a little early for his TZ [11:58:16] nuria: yeah, both are in the US timezone, IIRC [11:58:17] west coast [11:59:15] hello [11:59:38] hello hashar [12:12:11] greg-g: https://wikitech.wikimedia.org/wiki/Deployments#Next_month => July 3 is next week :) [12:38:43] RECOVERY - DPKG on osmium is OK: All packages OK [12:53:45] (03PS1) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142517 [12:54:23] (03Abandoned) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142517 (owner: 10Hashar) [13:15:46] quiet day huh [13:17:47] I need one [13:20:41] chasemp: ping [13:20:57] root@copper:~# groups [13:21:02] root wikidev [13:21:02] this is... just wrong :) [13:23:39] RECOVERY - Unmerged changes on repository puppet on strontium is OK: Fetching origin [13:25:05] hashar: Do we have a bug report for all those weird merge failures in gerrit/zuul? [13:25:10] See https://gerrit.wikimedia.org/r/#/c/141943/ [13:25:20] I explicitly do a rebase and then zuul/jenkins-bot says it can't be merged [13:25:21] Krinkle: yup against gerrit [13:25:25] search for hash mismatch [13:25:29] Maybe it's a few seconds delayed? [13:25:33] (race condition unknown hash?) [13:25:38] got a link? [13:25:42] work around? [13:26:00] work around: try again :D [13:26:04] ok [13:26:43] Krinkle: in Zuul the merge operations is now done by a dedicated process "zuul-merger" [13:27:00] it only runs on gallium now and logs error to /var/log/zuul/merger.log [13:27:07] let me paste [13:27:15] hashar: How was it done previously, and what is the advantage of that? [13:27:20] http://paste.debian.net/107019/ [13:27:27] GitCommandError: 'git remote update origin' returned exit status 1: hash mismatch [13:27:27] key_verify failed for server_host_key [13:27:31] that is an issue in Gerrit [13:28:10] the merge operation is blocking the scheduler and takes a bunch of resources + its quite slow [13:28:28] so openstack extracted the merge operations to a client and have several mergers [13:28:33] that largely speed up their operations [13:28:45] in our case, I think a merge of mw/core takes 20 - 25 seconds [13:28:47] and it's async from zuul's main event loop? [13:28:54] I can't remember whether it is async [13:28:56] I think so [13:29:08] by merge you mean simulated merge against lastest master for testing, not CR+2 merge right? [13:29:16] (that's done by gerrit afaik) [13:29:21] probably the scheduler send a merge request and keep proceeding other things [13:29:31] cool [13:29:32] then the merger report it finished it is work and the scheduler resume and then trigger jobs [13:29:36] that is hopefully how it works [13:29:39] Yeah [13:29:57] the CR+2 triggers a merge of the patchset against the tip of master as well :] [13:30:01] Is this 'merge failed' error new, or did it happen before as well? [13:30:02] (03CR) 10Alexandros Kosiaris: [C: 032] Sanitize nrpe::check title parameter [operations/puppet] - 10https://gerrit.wikimedia.org/r/141951 (owner: 10Alexandros Kosiaris) [13:30:21] or, if it is the second patchset in the gate pipeline, a merge of that second patchset on top of master + previous patchset in the queue [13:30:35] and the hash mismatch error is quite old [13:30:39] a few months old [13:30:53] do you have a link to the upstream bug report? [13:30:55] there is not that many occurrences. Maybe two or three per days in Zuul [13:31:18] Me, myself and I would like to upvote it [13:31:24] https://bugzilla.wikimedia.org/show_bug.cgi?id=53895 (hash-mismatch) Gerrit SSH: Intermittent key_verify failed for server_host_key and 'hash mismatch' [13:31:27] (and CC) [13:31:34] been opened on Sept 2013 [13:31:51] OpenStack is hit by that issue as well [13:32:06] QChris talked about it with OpenStack folks as well but we haven't found anything [13:32:34] Is there an issue in the bug tracker of gerit? [13:32:35] gerrt [13:33:08] Looks like there's a possible fix [13:33:09] https://github.com/apache/mina-sshd/pull/5 [13:33:33] https://issues.apache.org/jira/browse/SSHD-330 [13:33:48] It was merged! [13:37:20] Krinkle: copy paste that to the bug :-] [13:37:48] maybe qchris can comment on it / try it out [13:39:01] I did, 2 minutes ago [13:41:56] thx [13:52:18] ^d: fyi, I've approached GitHub support about getting a 'mirror' flag for the repos we have on github that are mirrors [13:52:25] Like https://github.com/apache has now as well. [13:52:54] Paravoid what's up [13:52:55] I've asked them to not do anything yet due to how we use Gerrit and also PR-syncing on some repos, but just to see what's up and what is possible [14:06:38] chasemp: root@copper:~# groups [14:06:40] chasemp: root wikidev [14:08:18] The members logic makes root the default member of a group specified but without other members [14:10:36] (03PS7) 10Ottomata: Have Wikimetrics use the redis module's configuration again [operations/puppet] - 10https://gerrit.wikimedia.org/r/141120 (https://bugzilla.wikimedia.org/66911) (owner: 10QChris) [14:11:47] yeah that's wrong [14:11:54] If you were to add user foo to wikidev root without being specified would be left out, anyways that's the logic we went with. Not against changing it. [14:12:01] root is definitely not wikidev :) [14:14:14] (03CR) 10Ottomata: [C: 032 V: 032] Have Wikimetrics use the redis module's configuration again [operations/puppet] - 10https://gerrit.wikimedia.org/r/141120 (https://bugzilla.wikimedia.org/66911) (owner: 10QChris) [14:14:20] (03CR) 10Ottomata: [C: 032 V: 032] Add backup role and scripts to wikimetrics [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/139557 (https://bugzilla.wikimedia.org/66119) (owner: 10Milimetric) [14:15:54] (03PS11) 10Ottomata: Enable the new backup role in wikimetrics if set [operations/puppet] - 10https://gerrit.wikimedia.org/r/139558 (https://bugzilla.wikimedia.org/66119) (owner: 10Milimetric) [14:16:10] (03CR) 10Ottomata: [C: 032 V: 032] Enable the new backup role in wikimetrics if set [operations/puppet] - 10https://gerrit.wikimedia.org/r/139558 (https://bugzilla.wikimedia.org/66119) (owner: 10Milimetric) [14:18:27] !log Updated our Jenkins Job Builder fork: e9db73d..0972985 [14:18:32] Logged the message, Master [14:18:40] I think it was gpasswd won't accept an empty member string so if you remove people and bail on empty before the op it leaves stragglers behind and if you try to specify no members it errors so this was the compromise. But hopefully there is a better solution. Anyhoo off to get ready for the day [14:22:35] (03CR) 10Andrew Bogott: [C: 032] graphite: Remove auth requirement for labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/142494 (owner: 10Yuvipanda) [14:23:22] andrewbogott: woot! ty [14:24:56] (03CR) 10Andrew Bogott: [C: 04-1] "One inline comment, maybe a result of the rebase" (031 comment) [operations/puppet/nginx] - 10https://gerrit.wikimedia.org/r/142499 (https://bugzilla.wikimedia.org/67188) (owner: 10Yuvipanda) [14:25:27] andrewbogott: see my comment on the patch as well. [14:25:36] andrewbogott: I don't know why -debug is needed there, the commit message didn't explain. [14:26:06] akosiaris: it looks to me liike your puppet icinga test from yesterday isn't getting run anywhere. Is that on purpose? [14:26:32] Yesterday mutante added a patch to set up the 'file' component of the test but we're still clearly missing a bit. [14:28:12] (03PS1) 10Ottomata: Update README with CDH5 docs [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/142529 [14:29:01] (03PS2) 10Ottomata: Update README with CDH5 docs [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/142529 [14:29:09] andrewbogott: no it is not on purpose. It should be running everywhee [14:29:15] s/ee/ere/ [14:29:29] akosiaris: do you see it on the icinga panel? Maybe I'm looking for the wrong thing. [14:29:45] (03PS1) 10Ottomata: Update README with CDH5 docs [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142530 [14:29:51] (03PS2) 10Scottlee: Fixed spacing and lint rules for manifests files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142479 [14:30:01] (03Abandoned) 10Ottomata: Update README with CDH5 docs [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/142529 (owner: 10Ottomata) [14:30:06] nope [14:30:13] (03CR) 10Ottomata: [C: 032 V: 032] Update README with CDH5 docs [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142530 (owner: 10Ottomata) [14:31:07] does it need to be defined in checkcommands.cfg.erb? [14:31:09] (03CR) 10jenkins-bot: [V: 04-1] Fixed spacing and lint rules for manifests files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142479 (owner: 10Scottlee) [14:31:19] no [14:31:28] it is an nrpe service [14:31:37] so all it needs is nrpe::monitor_service stanza [14:31:56] which I already got. But for some reason icinga does not have those populated [14:32:00] looking into it now [14:32:11] 'k [14:32:32] * YuviPanda pokes akosiaris to see if there was any progress on postgres [14:32:50] prostgres [14:33:08] marktraceur: contgress [14:33:08] progres? /me gives up [14:33:22] postmanteau [14:34:05] postman tao [14:34:07] YuviPanda: We are about to move the machines and equip them with procured SSDs. Got RT access ? I can give you the tickets to watch for if you feel like it [14:34:23] akosiaris: sure! I have RT access, don't know if it has access to that particular queue. [14:35:34] YuviPanda: https://rt.wikimedia.org/Ticket/Display.html?id=7591 [14:36:58] ori: So, I wasn't around for this, which may answer my question, but why didn't https://gerrit.wikimedia.org/r/138634 get merged and deployed during the SWAT? [14:37:12] Too late now but #retrospectivetime [14:37:13] (03PS2) 10Scottlee: Fixed spacing and lint rules for manifests/misc files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142483 [14:37:41] akosiaris: ty [14:38:30] (03CR) 10jenkins-bot: [V: 04-1] Fixed spacing and lint rules for manifests/misc files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142483 (owner: 10Scottlee) [14:38:46] can someone tell me why these failed the test build? https://gerrit.wikimedia.org/r/#/c/142479/ https://gerrit.wikimedia.org/r/#/c/142483/ [14:38:59] the errors keep referring to /manifests/role/swift.pp which aren't even edited in these commits [14:39:25] I worked so hard on these...... :( [14:39:59] * marktraceur looks [14:42:42] dogeydogey: no they don't. they refer to manifests/swift.pp at least and there is an error line next to it like manifests/swift.pp:207 [14:43:07] and looking at https://integration.wikimedia.org/ci/job/operations-puppet-pplint-HEAD/56/console seems like there are 3 different errors at least [14:43:31] akosiaris yeah i just fixed those and pushed a second build [14:43:34] but still failing [14:44:13] or they are errors that are were there before I did any modifications [14:44:22] i suggest you install puppet on your machine and run puppet parser validate on every file you modify [14:44:30] before pushing to gerrit. It will save you time [14:44:51] (03PS1) 10Yuvipanda: toollabs: Enable diamond monitoring for Redis [operations/puppet] - 10https://gerrit.wikimedia.org/r/142531 [14:44:57] andrewbogott: ^ enables diamond collection for redis [14:45:44] chasemp: ^ as well. [14:48:59] (03CR) 10Andrew Bogott: [C: 032] toollabs: Enable diamond monitoring for Redis [operations/puppet] - 10https://gerrit.wikimedia.org/r/142531 (owner: 10Yuvipanda) [14:49:07] andrewbogott: ty [14:51:20] akosiaris that's what I did.... [14:51:27] I couldn't solve every single ones [14:51:34] sometimes there are hundreds [14:52:12] hundreds of syntax errors ? [14:57:04] dogeydogey: you might have an easier time if you pare your patch into a bunch of smaller ones, like maybe one module at a time. [14:58:26] andrewbogott is there any way to salvage the ones I've committed? because that's what I did yesterday when we worked on my merge issue, it was just everything in the manifest folder, but then I broke it down into three chunks, two of which I have committed and are not passing tests [14:58:53] (03PS1) 10Yuvipanda: toollabs: Install python-redis module for diamond collector [operations/puppet] - 10https://gerrit.wikimedia.org/r/142533 [14:59:11] andrewbogott: ^ again, apparently the diamond module needs that package. I installed it manually and it works, so this is just puppetizing status quo [14:59:51] dogeydogey: I'm not sure I understand the question :) You can certainly salvage the changes and just rearrange them into different smaller patches. [15:00:13] (03CR) 10Andrew Bogott: [C: 032] toollabs: Install python-redis module for diamond collector [operations/puppet] - 10https://gerrit.wikimedia.org/r/142533 (owner: 10Yuvipanda) [15:00:31] andrewbogott: ty [15:07:51] @notify ori [15:07:51] This user is now online in #wikimedia-dev. I'll let you know when they show some activity (talk, etc.) [15:10:54] He's...in this channel too, YuviPanda [15:10:56] What you smoking [15:11:10] marktraceur: Asking for ori's timezone is pointless, so might as well automate that [15:11:17] Heh [15:11:24] YuviPanda: CTCP TIME obvs [15:11:31] Because that will be helpful. [15:11:44] * marktraceur still responds to it with UTC-7, e.g., despite being in -4 [15:11:47] marktraceur: CTCP's UX isn't great, you know. Won't use as a disincentive for CTCP devs. [15:11:58] Hah [15:12:07] What UX, it's a command [15:12:16] <^d> Timezones are lame. [15:15:13] * hashar relocates everyone to CET [15:15:46] marktraceur: command line arguments most certainly have UI/UX [15:15:57] marktraceur: bzr's UI is/was WAY better than git's [15:16:13] how you name commands etc is important [15:16:18] Heh [15:16:30] hashar: Yes because UTC+1 (2?) is obviously the answer [15:16:45] where can I find CTCP's Trello board? [15:16:57] * marktraceur kickbans YuviPanda for trolling [15:17:11] Trelling? [15:19:11] marktraceur: :) [15:24:40] manybubbles|away: when you get back [15:24:49] just double checking that I should add the new elastic nodes to pybal [15:28:29] ottomata: sure [15:29:56] ok! [15:30:23] adding. [15:30:38] http://noc.wikimedia.org/pybal/eqiad/search [15:30:40] ees good? [15:30:54] (03PS1) 10Scottlee: Fixed spacing and puppet-lint issues in manifests/role. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142542 [15:31:20] ottomata: looks good [15:35:18] cooool [15:35:42] ^demon|away: I submitted https://gerrit.wikimedia.org/r/140890 multiple times sans effect, is there some issue with Gerrit right now? [15:35:44] (03PS1) 10Scottlee: Fixed puppet-lint issues on manifests/role/analytics/hadoop.pp. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142543 [15:43:18] (03CR) 10Dzahn: [C: 031] Fixed puppet-lint issues on manifests/role/analytics/hadoop.pp. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142543 (owner: 10Scottlee) [15:46:42] (03PS1) 10Alexandros Kosiaris: Split base::puppet to it own file. Amend puppet freshness check [operations/puppet] - 10https://gerrit.wikimedia.org/r/142553 [15:46:44] (03PS1) 10Alexandros Kosiaris: Puppet 3.x needs storeconfigs=true for thin_storeconfigs to work [operations/puppet] - 10https://gerrit.wikimedia.org/r/142554 [15:47:32] akosiaris: what is missing that makes the new puppet check not end up in Icinga config yet? [15:47:37] andrewbogott: found it. https://gerrit.wikimedia.org/r/142554 [15:47:40] mutante: ^ [15:47:44] this [15:48:19] akosiaris: oh, so if we were still on puppet 2.x , this would have already showed up after i added the file{} resource? [15:48:38] wow, wouldn't have found that one soon [15:48:47] akosiaris: wouldn't that just make the test fail? Rather than not registering at all? [15:49:20] mutante: had nothing to do with the file resource (which was needed however indeed, thanks for catching that) [15:49:35] andrewbogott: the exported resource never got in the database [15:49:39] akosiaris: yea, i know, but i mean, that _would_ have been all if it wasn't puppet 3 [15:49:56] so, it never got populated on neon [15:49:57] oh, ah [15:50:05] akosiaris: Ah, so missing that meant no new tests ever, wasn't specific to this test? [15:50:06] mutante: well seems like it [15:50:12] andrewbogott: yes [15:50:14] (03PS2) 10BBlack: varnish (3.0.5plus~x-wm6) unstable; urgency=low [operations/debs/varnish] (3.0.5-plus-wm) - 10https://gerrit.wikimedia.org/r/142283 [15:50:26] merging, restarting and seeing what changed now [15:50:38] akosiaris: ok, obscure but makes sense [15:50:55] (03CR) 10Alexandros Kosiaris: [C: 032] Puppet 3.x needs storeconfigs=true for thin_storeconfigs to work [operations/puppet] - 10https://gerrit.wikimedia.org/r/142554 (owner: 10Alexandros Kosiaris) [15:51:50] (03CR) 10Alexandros Kosiaris: [C: 032] Split base::puppet to it own file. Amend puppet freshness check [operations/puppet] - 10https://gerrit.wikimedia.org/r/142553 (owner: 10Alexandros Kosiaris) [15:52:04] so it also didnt influence any existing check because they were already in the db [15:52:14] therefore we could have compared it to the other ones forever... [15:52:39] :-( [15:52:46] damn puppet .... [15:52:55] ?doest work yet? [15:53:13] niah, it was just a more verbose sigh [15:53:15] testing now [15:53:37] 'k:) [15:55:43] <^demon|away> marktraceur: Check the actual clone locally. Sometimes the UI lies and it actually merged. [15:56:56] (03PS1) 10Alexandros Kosiaris: Fix typo in warning vs critical [operations/puppet] - 10https://gerrit.wikimedia.org/r/142557 [15:57:19] Heh. [15:57:33] (03CR) 10Dzahn: [C: 032] Fix typo in warning vs critical [operations/puppet] - 10https://gerrit.wikimedia.org/r/142557 (owner: 10Alexandros Kosiaris) [15:58:25] ^demon|away: No dice [15:58:28] (03CR) 10Alexandros Kosiaris: [V: 032] Fix typo in warning vs critical [operations/puppet] - 10https://gerrit.wikimedia.org/r/142557 (owner: 10Alexandros Kosiaris) [15:59:06] 40 mins and it should be everywhere [15:59:18] :) [16:00:12] So I accidentally merged to wmf10 - https://gerrit.wikimedia.org/r/141300 - should I revert or just note it somewhere? I want it to go out on Monday's first SWAT anyway... [16:00:42] <^demon|away> Well the submodule ref on mw/core won't update automatically. [16:01:56] (03PS1) 10Ottomata: Put CDH 5.0.2 in apt updates [operations/puppet] - 10https://gerrit.wikimedia.org/r/142558 [16:01:57] Ah, right. [16:02:18] * marktraceur notes this for Monday [16:02:35] akosiaris: https://gerrit.wikimedia.org/r/#/c/142558/1/modules/install-server/files/reprepro/updates quick +1? [16:04:32] ottomata: hcatalog no more ? [16:05:59] (03PS1) 10Dzahn: rm old puppet_disabled check,replaced by new chk [operations/puppet] - 10https://gerrit.wikimedia.org/r/142560 [16:06:13] its renamed to hive [16:06:16] so i think it shoudl matc [16:06:18] hive-hcatalog [16:06:20] part of hive suite now [16:07:08] ok [16:07:23] (03CR) 10Alexandros Kosiaris: [C: 032] Put CDH 5.0.2 in apt updates [operations/puppet] - 10https://gerrit.wikimedia.org/r/142558 (owner: 10Ottomata) [16:07:31] hmmm I merged [16:07:36] damn habbit [16:07:52] ottomata: want me to puppet-merge too ? [16:08:00] ja that's cool [16:08:01] danke [16:08:26] Error: Could not find any hostgroup matching 'virt_eqiad' (config file '/etc/icinga/puppet_hosts.cfg', starting on line 9535) [16:08:27] grrrrr [16:10:39] $nagios_group = ? [16:14:52] akosiaris: where did it get that group name from? [16:15:14] it is in the database but I haven't figure out why yet [16:16:18] ^demon|away: https://gerrit.wikimedia.org/r/140890 tried resubmitting, still no luck [16:16:18] akosiaris: default => "${cluster}_${::site}" [16:16:29] Oh FFS it has a dependency [16:16:32] * marktraceur stops being stupid [16:16:42] akosiaris: so it checks if something is cluster "misc" and then it has no hostgroups, otherwise .. the above [16:17:01] i think we need to just make the hostgroup [16:17:05] aaaahh nice [16:17:09] cool! nice work [16:17:19] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: Fetching origin [16:17:28] you are absolutely right of course [16:17:36] <^demon|away> marktraceur: it's friday, you're excused :) [16:17:38] let's see _where_ we add that ,though [16:17:39] again ? what is it with strontium these days ? [16:18:18] ^d: Also contributing, sub-par amounts of coffee, 20-hour drive yesterday [16:19:19] RECOVERY - Unmerged changes on repository puppet on strontium is OK: Fetching origin [16:20:19] (03PS1) 10Dzahn: add missing nagios hostgroup for virt eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/142562 [16:21:04] (03PS2) 10Dzahn: add missing nagios hostgroup for virt eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/142562 [16:21:22] akosiaris: ^ [16:21:24] mutante: why not the other virt hsots as well? [16:21:56] mutante: niah that is not correct [16:22:35] look at the end of manifests/misc/icinga.pp [16:22:42] it needs a line like this [16:22:47] question is where to put it [16:23:10] that is true, and .. a role ? [16:23:11] akosiaris: , q [16:23:16] http://archive-primary.cloudera.com/cdh5/ubuntu/precise/amd64/cdh/pool/contrib/p/ [16:23:24] do those need to be separate matches in the updates file? [16:23:31] specifically pig-udf-datafu [16:23:43] even though pig is present...i guess. [16:23:48] (03PS3) 10Andrew Bogott: add missing nagios hostgroup for virt eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/142562 (owner: 10Dzahn) [16:23:51] ottomata: only if they come from different source packages [16:23:53] grep-dctrl just matches these directory names exactly? [16:23:55] hmm [16:23:57] look at debian/control for pig [16:24:15] it might have it inside [16:24:19] well, i know that pig-udf-datafu wasn't included in those updates [16:24:22] that we just added :p [16:24:49] (03PS1) 10Dzahn: add global nagios hostgroup for virt hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/142564 [16:24:53] ottomata: http://archive-primary.cloudera.com/cdh5/ubuntu/precise/amd64/cdh/pool/contrib/p/pig-udf-datafu/pig-udf-datafu_0.0.4+cdh5.0.0+12-0.cdh5b1.p0.30.dsc [16:25:05] Source and Binary is what matters [16:25:16] so that line matches source [16:25:26] ah ok [16:25:28] so yes, needs more [16:25:29] awesome, danke [16:25:30] in order for the line to be well a little bill smaller [16:25:40] s/bill/bit/ [16:26:07] might be it is not worth it, but there are like source package with 10 binary packages [16:26:18] (03CR) 10Andrew Bogott: "This role file is generally applied on VMs, it doesn't contain anything that's applied on the prod hosts themselves. So this should go so" [operations/puppet] - 10https://gerrit.wikimedia.org/r/142564 (owner: 10Dzahn) [16:26:20] so you only include 1 and get 10 for free :-) [16:26:42] aye, these are regexes too, right? [16:26:50] i could just do ^pig.*$? [16:27:04] would you prefer I didn't do that? [16:27:09] and just manually listed all source packages? [16:27:56] the problem is you might match a package we already got [16:28:02] (03CR) 10Dzahn: "nevermind, we probably don't need this, it already does set the hostgroup to: < mutante> akosiaris: default => "${cluster}_${::site}" , we" [operations/puppet] - 10https://gerrit.wikimedia.org/r/142562 (owner: 10Dzahn) [16:28:32] and then we override and use the package from cloudera and not from us, ubuntu etc [16:28:35] hmm, ok [16:28:49] which is why I prefer to be a bit strict on that vs more lenient [16:28:49] (03Abandoned) 10Andrew Bogott: add missing nagios hostgroup for virt eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/142562 (owner: 10Dzahn) [16:28:51] (03PS1) 10Ottomata: Add pig-udf-datafu and parquet packages to apt updates [operations/puppet] - 10https://gerrit.wikimedia.org/r/142565 [16:28:52] aye [16:28:54] andrewbogott: sooo many labs roles ... [16:29:29] mutante: where are other similar groups defined? They're not all in one place somewhere? [16:29:49] (03PS2) 10Ottomata: Add pig-udf-datafu, avro-libs and parquet packages to apt updates [operations/puppet] - 10https://gerrit.wikimedia.org/r/142565 [16:30:15] (03PS3) 10Ottomata: Add pig-udf-datafu, avro-libs and parquet packages to apt updates [operations/puppet] - 10https://gerrit.wikimedia.org/r/142565 [16:30:21] andrewbogott: usually in role/foo.pp grep -r "monitor_group" * [16:30:21] (03CR) 10Ottomata: [C: 032 V: 032] Add pig-udf-datafu, avro-libs and parquet packages to apt updates [operations/puppet] - 10https://gerrit.wikimedia.org/r/142565 (owner: 10Ottomata) [16:30:39] mutante: ok, in that case nova.pp is the right place. [16:30:54] andrewbogott: mutante: strictly speaking this https://gerrit.wikimedia.org/r/#/c/142564/ is not in any role or class [16:31:15] it just happens to be in the file [16:31:28] ack, yea [16:31:32] it would work in almost any file that is import in site.pp [16:31:37] akosiaris: sure. But that file is still a weird place for it. It's the difference between 'in labs' and 'labs infra' [16:31:38] s/import/imported/ [16:31:59] andrewbogott: oh I see what you mean [16:32:15] ok so whatever file has labs infra and is imported from site.pp should be fine then [16:32:53] so any role file as we got import 'role/*.pp' in site.pp [16:33:40] (03PS2) 10Dzahn: add global nagios hostgroup for virt hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/142564 [16:33:42] there, nova [16:34:18] (03CR) 10Andrew Bogott: [C: 032] "better, thanks." [operations/puppet] - 10https://gerrit.wikimedia.org/r/142564 (owner: 10Dzahn) [16:34:57] hmm, akosiaris reprepro update is telling me nothing to do... [16:35:04] even thought it does to have things to do! [16:35:09] (03CR) 10Dzahn: [C: 032] "< akosiaris> so any role file as we got import 'role/*.pp' in site.pp" [operations/puppet] - 10https://gerrit.wikimedia.org/r/142564 (owner: 10Dzahn) [16:35:42] (03PS1) 10Alexandros Kosiaris: Remove zfs monitoring [operations/puppet] - 10https://gerrit.wikimedia.org/r/142569 [16:36:24] running with --noskipold... [16:36:54] hmm, i think that looks better [16:37:01] yup! [16:37:26] :-) [16:37:52] akosiaris: andrewbogott: ehem.. look how many hostgroups we have that don't have any hosts, btw :p https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?hostgroup=all&style=summary&nostatusheader [16:37:57] ahahaha [16:38:01] virt_pmtpa now [16:38:18] mutante: that's something that's throwing errors but didn't before? [16:38:44] andrewbogott: no, i think it's been like that for a long time and doesnt cause errors, it just doesnt do anything.. empty groups [16:38:59] ah, I see. [16:39:08] well, we could try populating them [16:39:27] but if we populated them.. we'd have nicer Icinga deep links [16:39:34] that show you the status of just one certain group [16:39:51] ok, but one by one.. we still have an error on puppet run on neon now [16:40:09] Error: Could not find any hostgroup matching 'virt_pmtpa' [16:40:16] yeah fixing now [16:40:17] lol, what you just said, ok [16:40:33] I am gonna put all DCs in there [16:40:55] If you populate a bunch of them now I might request that those entries all get consolidated in one file rather than being scattered among various roles [16:41:00] Although I guess I can see arguments for either approach [16:41:01] so at least it won't bite when we decide to have virt nodes in ulsfo,codfw or whatever else [16:41:07] +1 [16:41:12] (03PS1) 10Alexandros Kosiaris: Add the rest of the datacenter virt_ hostgroups [operations/puppet] - 10https://gerrit.wikimedia.org/r/142571 [16:41:28] like this andrewbogott ^ ? [16:41:33] or something else ? [16:42:02] are all labs servers also virt servers? [16:42:13] or are just "virt*" servers actual virt servers [16:42:20] just the terminology... [16:42:34] akosiaris: if we're only doing virt stuff then that's fine. If you're things for a bunch of other clusters as well, then… might be time for a reorg. [16:43:00] mutante: Right now we're just using 'virt' as a synonym for 'labs' [16:43:03] like labnet1001, that isn't called "virt", but it's also [16:43:10] The host names aren't especially obvious that way. [16:43:10] not today (Friday almost 8pm here) but yeah I get your point now [16:43:24] and labstore1001 right ? [16:43:27] In theory we could group all nova-compute hosts in one cluster and have the other labs stuff in a different one, in order to asses VM resource use [16:43:30] but, no need to do that today. [16:43:42] akosiaris: yeah [16:44:12] (03CR) 10Alexandros Kosiaris: [C: 032] Add the rest of the datacenter virt_ hostgroups [operations/puppet] - 10https://gerrit.wikimedia.org/r/142571 (owner: 10Alexandros Kosiaris) [16:44:27] (03CR) 10BBlack: [C: 032] varnish (3.0.5plus~x-wm6) unstable; urgency=low [operations/debs/varnish] (3.0.5-plus-wm) - 10https://gerrit.wikimedia.org/r/142283 (owner: 10BBlack) [16:46:30] labstore1001, disk space warning.. btw [16:46:31] (03PS1) 10Scottlee: Fixed puppet-lint issues [operations/puppet] - 10https://gerrit.wikimedia.org/r/142572 [16:46:33] andrewbogott: [16:46:55] /exp/dumps 0 MB [16:47:26] this is getting grrrr [16:47:30] ssl_esams hostgroup now [16:47:50] akosiaris: hence 'things for a bunch of other clusters' :) [16:47:50] (03CR) 10jenkins-bot: [V: 04-1] Fixed puppet-lint issues [operations/puppet] - 10https://gerrit.wikimedia.org/r/142572 (owner: 10Scottlee) [16:49:34] !log updated carbon repo varnish pkg to 3.0.5plus~x-wm6 [16:49:39] Logged the message, Master [16:52:57] (03Abandoned) 10Scottlee: Fixed spacing and lint rules for manifests/misc files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142483 (owner: 10Scottlee) [16:53:03] (03Abandoned) 10Scottlee: Fixed spacing and lint rules for manifests files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142479 (owner: 10Scottlee) [16:53:09] (03Abandoned) 10Scottlee: Fixed puppet-lint issues [operations/puppet] - 10https://gerrit.wikimedia.org/r/142572 (owner: 10Scottlee) [16:54:34] 42 rows in set (0.00 sec) [16:54:48] argh... the hostgroups already in the DB [16:54:54] never mind the possible ones... [16:55:21] (03PS1) 10Ottomata: Use mapreduce.shuffle instead of mapreduce_shuffle as comes with default yarn-site.xml [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142575 [16:55:36] (03CR) 10Ottomata: [C: 032 V: 032] Use mapreduce.shuffle instead of mapreduce_shuffle as comes with default yarn-site.xml [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142575 (owner: 10Ottomata) [16:56:13] something is wrong with jobqueue monitoring [16:56:19] UNKNOWN: No valid datapoints found [16:56:24] more than 0 jobs queued... [16:56:25] is there a way i can ask for a cached special page to be regenerated now as a one-time thing? [16:57:23] (03PS1) 10Ottomata: Updates for TODO.md [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142576 [16:57:38] (03CR) 10Ottomata: [C: 032 V: 032] Updates for TODO.md [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142576 (owner: 10Ottomata) [16:57:52] (03PS1) 10Ottomata: Use exec to make sure hosts.exclude exists rather than managing file resource [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142579 [16:58:22] (03CR) 10Ottomata: [C: 032 V: 032] Use exec to make sure hosts.exclude exists rather than managing file resource [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142579 (owner: 10Ottomata) [16:58:51] <^d> mutante: Just the monitoring, job queue itself looks ok. [16:59:17] ^d: thanks, good! [16:59:33] looks like it's been for at least 6d [17:02:38] chasemp: hey! [17:03:17] chasemp: so I'm trying to make the nginx module work with any package, https://gerrit.wikimedia.org/r/#/c/142499/2 [17:03:42] you rang....:) [17:04:25] tan4.corp.wikimedia.org ? :O [17:06:17] YuviPanda: my only thought would be what andrewbogott already asked there? [17:06:31] nginx is way more his ballgame especially considering ...submodule :) [17:08:35] chasemp: ah :) [17:08:47] chasemp: I'll just wait for ori then [17:09:34] (03PS1) 10Ottomata: Add TODO for fixing Oozie Server extjs install [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142581 [17:10:27] (03CR) 10Ottomata: [C: 032 V: 032] Add TODO for fixing Oozie Server extjs install [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142581 (owner: 10Ottomata) [17:16:46] (03PS1) 10Alexandros Kosiaris: Populate the rest of the nagios hostgroups [operations/puppet] - 10https://gerrit.wikimedia.org/r/142589 [17:17:06] (03PS1) 10Chad: Move about half of pool 4 lsearchd wikis to CirrusSearch [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142590 [17:19:02] mutante: andrewbogott https://gerrit.wikimedia.org/r/#/c/142589/ [17:19:05] a quick +1 ? [17:19:09] (03CR) 10Manybubbles: [C: 031] "I'm game to try it any time." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142590 (owner: 10Chad) [17:19:42] I went through with a for loop and got all possible hostgroups in the DB plus some other permutations. I hope this should fix it [17:19:55] (03CR) 10Andrew Bogott: [C: 031] Populate the rest of the nagios hostgroups [operations/puppet] - 10https://gerrit.wikimedia.org/r/142589 (owner: 10Alexandros Kosiaris) [17:20:36] (03CR) 10Dzahn: [C: 031] Populate the rest of the nagios hostgroups [operations/puppet] - 10https://gerrit.wikimedia.org/r/142589 (owner: 10Alexandros Kosiaris) [17:20:47] akosiaris: my favorite one is "pdf pmtpa" :p [17:21:07] when is that thing gonna die ???? [17:21:24] (03CR) 10Alexandros Kosiaris: [C: 032] Populate the rest of the nagios hostgroups [operations/puppet] - 10https://gerrit.wikimedia.org/r/142589 (owner: 10Alexandros Kosiaris) [17:21:31] (03PS2) 10Chad: Move about half of pool 4 lsearchd wikis to CirrusSearch [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142590 [17:22:05] akosiaris: nobody knows [17:22:39] akosiaris: 6149 final decisison / migration of PDF servers .. 8 months ago .. open [17:24:20] RECOVERY - Kafka Broker Replica Lag on analytics1021 is OK: kafka.server.ReplicaFetcherManager.Replica-MaxLag.Value OKAY: 0.0 [17:25:36] NRPE: Unable to read output [17:25:42] yey... :-( [17:25:49] at least it is finally running [17:26:20] PROBLEM - puppet last run on db1058 is CRITICAL: NRPE: Command check_puppet_checkpuppetrun not defined [17:26:20] PROBLEM - puppet last run on analytics1026 is CRITICAL: CRITICAL: Puppet last ran 1403889975 seconds ago, expected 14400 [17:26:20] PROBLEM - puppet last run on copper is CRITICAL: CRITICAL: Puppet last ran 1403889975 seconds ago, expected 14400 [17:26:21] PROBLEM - puppet last run on cp1054 is CRITICAL: CRITICAL: Puppet last ran 1403889975 seconds ago, expected 14400 [17:26:21] PROBLEM - puppet last run on analytics1014 is CRITICAL: CRITICAL: Puppet last ran 1403889975 seconds ago, expected 14400 [17:26:28] HEY! [17:26:31] hehehe [17:28:22] ok seems like it is due to a sudo thingy missing [17:28:51] I'm a little surprised that there are so few of those :) [17:29:35] still early [17:29:39] wait a couple of minutes :-) [17:29:47] andrewbogott: icinga bot just left for flooding.. who knows [17:30:17] 680 is what the interface says [17:36:30] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 1228 seconds ago with 0 failures [17:38:13] (03PS1) 10Alexandros Kosiaris: check_puppetrun needs sudo privs [operations/puppet] - 10https://gerrit.wikimedia.org/r/142592 [17:38:42] (03CR) 10Alexandros Kosiaris: [C: 032] check_puppetrun needs sudo privs [operations/puppet] - 10https://gerrit.wikimedia.org/r/142592 (owner: 10Alexandros Kosiaris) [17:38:51] (03CR) 10Alexandros Kosiaris: [V: 032] check_puppetrun needs sudo privs [operations/puppet] - 10https://gerrit.wikimedia.org/r/142592 (owner: 10Alexandros Kosiaris) [17:39:37] this should be the last fix. In max 20 mins everything should be ok in icinga's console [17:41:17] (03PS1) 10Dzahn: add legalpad.wm.org as CNAME for radon [operations/dns] - 10https://gerrit.wikimedia.org/r/142593 [17:42:17] (03CR) 10Rush: [C: 031] "nice" [operations/dns] - 10https://gerrit.wikimedia.org/r/142593 (owner: 10Dzahn) [17:42:42] (03CR) 10Dzahn: [C: 032] add legalpad.wm.org as CNAME for radon [operations/dns] - 10https://gerrit.wikimedia.org/r/142593 (owner: 10Dzahn) [17:47:09] akosiaris: feedback on https://gerrit.wikimedia.org/r/#/c/142400/ appreciated! [17:47:20] (03PS1) 10Dzahn: add varnish backend probe for legalpad [operations/puppet] - 10https://gerrit.wikimedia.org/r/142594 [17:47:46] hey ori! do you know why nginx-full-dbg is installed by default on nginx package? [17:47:59] PROBLEM - puppet last run on sanger is CRITICAL: CRITICAL: Puppet last ran 1403891271 seconds ago, expected 14400 [17:48:35] akosiaris: very nice, thanks [17:48:46] YuviPanda: debug symbols are nice [17:49:05] YuviPanda: whenever the topic has come up (as i recall, at least) everyone agreed we should install them where available [17:49:14] ori: hmm, true but causes problems for https://gerrit.wikimedia.org/r/#/c/142499/2 [17:49:16] why wouldn't you want it? [17:50:00] ori: I want to be able to customize which nginx package is installed. Want to use the nginx module for toollabs' dynamicproxy, mostly so I can get rid of the hand-management of the conf files there and also get the diamond collector for free. [17:50:03] YuviPanda: that's a very strange patch [17:50:18] ori: indeed, I was pretty sure it wasn't very useful (I -1'd it as well, I think) [17:50:30] what's the name of the package that you want to install? [17:51:08] ori: nginx-extras [17:51:21] ori: superset of nginx-full, but I am not sure if prod wants it [17:51:36] so in the dynamicproxy manifest [17:51:39] just do something like [17:51:57] package { 'nginx-extras': ensure => present } -> class { 'nginx': } [17:52:04] done [17:52:06] ah, hhmm. [17:52:12] I forgot that nginx-extras will replace nginx-full [17:52:14] rather than conflict [17:52:26] replace? [17:52:47] ori: as in, if you have nginx-full on a system, and install nginx-extras, nginx-full will be uninstalled by apt? [17:53:21] oh [17:53:55] indeed. apt-cache show nginx-extras | grep Conflicts [17:53:59] Conflicts: nginx-full, nginx-light, nginx-naxsi [17:54:00] the cleanest thing to do, then, is to just replace nginx-full / nginx-full-dbg in the nginx module with nginx-extra / nginx-extra-dbg, and to do the requisite legwork and asking around to make sure it's ok for prod [17:54:28] hmm. [17:54:41] I don't know how much work that'll be, considering the amount of stuff that comes with nginx-extras [17:54:49] also unsure if that'll have performance implications [17:54:57] ori: who do you think I should talk to? _joe_? [17:55:19] ori: what would be the issue with just packing the package basename be a param? [17:55:25] akosiari.s, paravoi.d, ryan lane, kartik [17:55:36] wait what [17:55:43] what's up? [17:55:46] hey paravoid [17:55:56] nginx module installs nginx-full [17:56:00] nginx-full conflicts with nginx-extra [17:56:08] YuviPanda wants nginx-extra for labs [17:56:34] to be more exact, labs already runs nginx-extra, I just want to use the nginx puppet module there as well instead of hand-managing the config with file {} directives in puppet [17:57:17] paravoid: so ori is suggesting running nginx-extras in prod as well, but I don't know how feasible that is, since it comes with a *lot* of stuff that prod will probably never need, and also potentially decreases performance/increases attack surface [17:57:18] YuviPanda: your patch won't work well; puppet will keep installing the packages on top of one another on every run [17:57:40] YuviPanda: i'm suggesting that that would be cleanest and that you should do the requisite investigation / asking around to see if it that could work! [17:57:43] ori: yeah, that's because I first did the patch, and then did the rebase, and the packages directive installing the debug symbols came with the rebase [17:58:03] ori: so I decided to -1 and wait and talk rather than fix the packages directive. [17:58:29] PROBLEM - puppet last run on palladium is CRITICAL: CRITICAL: Puppet last ran 1403891905 seconds ago, expected 14400 [17:58:54] extra modules not built by nginx core but debian packages into -extras: 'Auth PAM, Chunkin, DAV Ext, Echo, Embedded Lua, Fancy Index, HttpHeadersMore, HTTP Substitution Filter, http push, Nginx Development Kit, Upload Progress, Upstream Fair Queue' [18:00:53] If you stick with this approach, rather than having a free-form name-your-package parameter, I'd have a "variant" parameter that is restricted to either "full" or "extras". That makes it clear that it's the module's responsibility to manage the packages [18:00:53] (03PS1) 10Alexandros Kosiaris: Fix double parameter definition [operations/puppet] - 10https://gerrit.wikimedia.org/r/142595 [18:01:10] ori: ah, right. that sounds nice as well. I can rework the patch to do that. [18:02:40] if $variant !~ /^(full|extra|light)/ { fail("'variant' must be 'full', 'extra', or 'light' (got: '${variant}'). } [18:03:01] /^(full|extra|light)$/ even [18:04:14] and then: package { [ "nginx-${variant}", "nginx-${variant}-dbg" ]: ensure => present }, etc. [18:04:39] ori: right. will do. [18:05:30] that's a 'nginx-naxsi' variant too if you want to be complete [18:05:35] *there's a [18:06:00] (03CR) 10Alexandros Kosiaris: [C: 032] Fix double parameter definition [operations/puppet] - 10https://gerrit.wikimedia.org/r/142595 (owner: 10Alexandros Kosiaris) [18:07:09] RECOVERY - puppet last run on mc1012 is OK: OK: Puppet is currently enabled, last run 1175 seconds ago with 0 failures [18:07:29] RECOVERY - puppet last run on polonium is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [18:07:49] RECOVERY - puppet last run on db1043 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [18:07:49] RECOVERY - puppet last run on db1016 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [18:07:49] RECOVERY - puppet last run on pc1002 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:07:49] RECOVERY - puppet last run on mw1076 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [18:07:49] RECOVERY - puppet last run on mw1149 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [18:07:50] RECOVERY - puppet last run on labstore1001 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [18:07:52] (03PS1) 10Dzahn: switch legalpad.wm over to misc varnish [operations/dns] - 10https://gerrit.wikimedia.org/r/142597 [18:07:59] RECOVERY - puppet last run on db1052 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [18:08:09] RECOVERY - puppet last run on db1003 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:08:09] RECOVERY - puppet last run on dataset1001 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [18:08:09] RECOVERY - puppet last run on mc1005 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:08:19] RECOVERY - puppet last run on analytics1026 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [18:08:19] RECOVERY - puppet last run on silver is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:08:29] RECOVERY - puppet last run on db1036 is OK: OK: Puppet is currently enabled, last run 1111 seconds ago with 0 failures [18:08:29] RECOVERY - puppet last run on snapshot1001 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:08:29] RECOVERY - puppet last run on es1007 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [18:08:29] RECOVERY - puppet last run on nescio is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [18:08:29] RECOVERY - puppet last run on mw1079 is OK: OK: Puppet is currently enabled, last run 1174 seconds ago with 0 failures [18:08:30] RECOVERY - puppet last run on elastic1006 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [18:08:30] RECOVERY - puppet last run on wtp1012 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:08:39] RECOVERY - puppet last run on mw1055 is OK: OK: Puppet is currently enabled, last run 1176 seconds ago with 0 failures [18:08:39] RECOVERY - puppet last run on amssq34 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [18:08:39] RECOVERY - puppet last run on cp1050 is OK: OK: Puppet is currently enabled, last run 1200 seconds ago with 0 failures [18:08:39] RECOVERY - puppet last run on virt1003 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [18:08:39] RECOVERY - puppet last run on mw1206 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [18:08:40] RECOVERY - puppet last run on db1026 is OK: OK: Puppet is currently enabled, last run 1207 seconds ago with 0 failures [18:08:40] RECOVERY - puppet last run on wtp1005 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [18:08:49] RECOVERY - puppet last run on labnet1001 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [18:08:49] RECOVERY - puppet last run on antimony is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [18:08:49] RECOVERY - puppet last run on db1060 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:08:49] RECOVERY - puppet last run on analytics1016 is OK: OK: Puppet is currently enabled, last run 1287 seconds ago with 0 failures [18:08:49] RECOVERY - puppet last run on cp3010 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:08:50] RECOVERY - puppet last run on gadolinium is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:08:50] RECOVERY - puppet last run on mw1151 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:08:51] RECOVERY - puppet last run on mw1162 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [18:08:51] RECOVERY - puppet last run on ssl1005 is OK: OK: Puppet is currently enabled, last run 1185 seconds ago with 0 failures [18:08:52] RECOVERY - puppet last run on ms-be3002 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:08:52] RECOVERY - puppet last run on osm-db1001 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [18:08:59] RECOVERY - puppet last run on mw1195 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [18:08:59] oh go :) [18:08:59] RECOVERY - puppet last run on search1002 is OK: OK: Puppet is currently enabled, last run 1197 seconds ago with 0 failures [18:08:59] RECOVERY - puppet last run on mw1168 is OK: OK: Puppet is currently enabled, last run 1192 seconds ago with 0 failures [18:09:01] god [18:09:09] RECOVERY - puppet last run on mw1014 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:09:09] RECOVERY - puppet last run on lvs4003 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [18:09:09] RECOVERY - puppet last run on db1039 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [18:09:09] RECOVERY - puppet last run on analytics1013 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:09:10] RECOVERY - puppet last run on mw1049 is OK: OK: Puppet is currently enabled, last run 1213 seconds ago with 0 failures [18:09:10] RECOVERY - puppet last run on mw1044 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [18:09:10] RECOVERY - puppet last run on mw1098 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [18:09:19] RECOVERY - puppet last run on cp4001 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:09:19] RECOVERY - puppet last run on elastic1019 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [18:09:19] RECOVERY - puppet last run on argon is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [18:09:19] RECOVERY - puppet last run on wtp1018 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [18:09:19] RECOVERY - puppet last run on mw1125 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [18:09:29] RECOVERY - puppet last run on mw1133 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:09:29] RECOVERY - puppet last run on db1069 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:09:29] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [18:09:29] RECOVERY - puppet last run on mw1202 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:09:29] RECOVERY - puppet last run on mw1050 is OK: OK: Puppet is currently enabled, last run 1094 seconds ago with 0 failures [18:09:30] PROBLEM - puppet last run on ms1001 is CRITICAL: CRITICAL: Puppet last ran 1403892564 seconds ago, expected 14400 [18:09:30] RECOVERY - puppet last run on mc1014 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:09:31] RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:09:31] RECOVERY - puppet last run on mw1051 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [18:09:32] RECOVERY - puppet last run on snapshot1002 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:09:39] RECOVERY - puppet last run on ssl1009 is OK: OK: Puppet is currently enabled, last run 1175 seconds ago with 0 failures [18:09:39] RECOVERY - puppet last run on mw1146 is OK: OK: Puppet is currently enabled, last run 1167 seconds ago with 0 failures [18:09:39] RECOVERY - puppet last run on cp4019 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [18:09:39] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:09:39] RECOVERY - puppet last run on amssq40 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [18:09:40] RECOVERY - puppet last run on cp1046 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [18:09:40] RECOVERY - puppet last run on mw1208 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [18:09:41] RECOVERY - puppet last run on mw1084 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [18:09:41] RECOVERY - puppet last run on mw1111 is OK: OK: Puppet is currently enabled, last run 1247 seconds ago with 0 failures [18:09:49] RECOVERY - puppet last run on db1020 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [18:09:49] RECOVERY - puppet last run on ms-be1008 is OK: OK: Puppet is currently enabled, last run 1185 seconds ago with 0 failures [18:09:49] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 1196 seconds ago with 0 failures [18:09:49] RECOVERY - puppet last run on db1071 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [18:09:50] RECOVERY - puppet last run on mw1180 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [18:09:50] RECOVERY - puppet last run on analytics1022 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [18:09:50] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 1195 seconds ago with 0 failures [18:09:51] RECOVERY - puppet last run on search1005 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [18:09:59] RECOVERY - puppet last run on virt1004 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [18:09:59] RECOVERY - puppet last run on amssq51 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:10:09] PROBLEM - puppet last run on dataset2 is CRITICAL: CRITICAL: Puppet last ran 1403892599 seconds ago, expected 14400 [18:10:09] RECOVERY - puppet last run on db1062 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:10:09] PROBLEM - puppet last run on ms1002 is CRITICAL: CRITICAL: Puppet last ran 1403892601 seconds ago, expected 14400 [18:10:09] RECOVERY - puppet last run on ms-be1009 is OK: OK: Puppet is currently enabled, last run 1136 seconds ago with 0 failures [18:10:09] PROBLEM - puppet last run on ms1004 is CRITICAL: CRITICAL: Puppet last ran 1403892605 seconds ago, expected 14400 [18:10:10] RECOVERY - puppet last run on rdb1001 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [18:10:10] RECOVERY - puppet last run on oxygen is OK: OK: Puppet is currently enabled, last run 1208 seconds ago with 0 failures [18:10:11] PROBLEM - puppet last run on es4 is CRITICAL: CRITICAL: Puppet last ran 1403892607 seconds ago, expected 14400 [18:10:11] RECOVERY - puppet last run on cp4005 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:10:12] RECOVERY - puppet last run on ssl3002 is OK: OK: Puppet is currently enabled, last run 1199 seconds ago with 0 failures [18:10:12] RECOVERY - puppet last run on amssq56 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [18:10:19] RECOVERY - puppet last run on hafnium is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [18:10:19] PROBLEM - puppet last run on linne is CRITICAL: CRITICAL: Puppet last ran 1403892612 seconds ago, expected 14400 [18:10:19] RECOVERY - puppet last run on search1017 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [18:10:21] /last PROBLEM [18:10:29] RECOVERY - puppet last run on osmium is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 9 failures [18:10:29] RECOVERY - puppet last run on mw1165 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [18:10:29] RECOVERY - puppet last run on rhenium is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:10:29] RECOVERY - puppet last run on virt1001 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [18:10:29] RECOVERY - puppet last run on mw1034 is OK: OK: Puppet is currently enabled, last run 1158 seconds ago with 0 failures [18:10:30] RECOVERY - puppet last run on analytics1023 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:10:30] RECOVERY - puppet last run on mw1188 is OK: OK: Puppet is currently enabled, last run 1162 seconds ago with 0 failures [18:10:31] PROBLEM - puppet last run on aluminium is CRITICAL: CRITICAL: Puppet last ran 1403892625 seconds ago, expected 14400 [18:10:31] RECOVERY - puppet last run on mw1171 is OK: OK: Puppet is currently enabled, last run 1217 seconds ago with 0 failures [18:10:32] RECOVERY - puppet last run on cp1062 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [18:10:32] RECOVERY - puppet last run on mw1056 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [18:10:33] RECOVERY - puppet last run on db72 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [18:10:33] RECOVERY - puppet last run on virt1007 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:10:39] RECOVERY - puppet last run on mc1001 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [18:10:39] RECOVERY - puppet last run on mw1057 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:10:39] RECOVERY - puppet last run on ms-be1012 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:10:39] RECOVERY - puppet last run on mw1116 is OK: OK: Puppet is currently enabled, last run 1166 seconds ago with 0 failures [18:10:39] RECOVERY - puppet last run on mw1156 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:10:40] RECOVERY - puppet last run on rubidium is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:10:40] RECOVERY - puppet last run on mw1159 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:10:41] RECOVERY - puppet last run on mw1004 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:10:41] RECOVERY - puppet last run on thallium is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [18:10:42] RECOVERY - puppet last run on cp1063 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [18:10:42] RECOVERY - puppet last run on snapshot1004 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [18:10:49] RECOVERY - puppet last run on db1004 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [18:10:49] RECOVERY - puppet last run on elastic1015 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [18:10:49] RECOVERY - puppet last run on cp1038 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:10:49] RECOVERY - puppet last run on elastic1014 is OK: OK: Puppet is currently enabled, last run 1181 seconds ago with 0 failures [18:10:49] RECOVERY - puppet last run on mw1183 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [18:10:50] RECOVERY - puppet last run on amssq36 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [18:10:50] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [18:10:51] icinga doesnt get kicked for flooding, wee :) [18:10:51] RECOVERY - puppet last run on cp4018 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [18:10:51] RECOVERY - puppet last run on mw1198 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [18:10:56] RECOVERY - puppet last run on mw1081 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [18:10:56] RECOVERY - puppet last run on db1055 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [18:10:56] PROBLEM - puppet last run on nickel is CRITICAL: CRITICAL: Puppet last ran 1403892645 seconds ago, expected 14400 [18:10:56] RECOVERY - puppet last run on db69 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [18:10:59] RECOVERY - puppet last run on mc1013 is OK: OK: Puppet is currently enabled, last run 1190 seconds ago with 0 failures [18:10:59] RECOVERY - puppet last run on hooft is OK: OK: Puppet is currently enabled, last run 1115 seconds ago with 0 failures [18:10:59] RECOVERY - puppet last run on mw1030 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:10:59] RECOVERY - puppet last run on search1024 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [18:10:59] PROBLEM - DPKG on osmium is CRITICAL: DPKG CRITICAL dpkg reports broken packages [18:11:00] RECOVERY - puppet last run on mw1097 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:11:09] RECOVERY - puppet last run on cp1048 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [18:11:09] RECOVERY - puppet last run on wtp1015 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [18:11:09] RECOVERY - puppet last run on wtp1022 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [18:11:09] RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [18:11:09] RECOVERY - puppet last run on mw1181 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [18:11:10] RECOVERY - puppet last run on mw1032 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [18:11:10] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [18:11:11] RECOVERY - puppet last run on wtp1002 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [18:11:19] RECOVERY - puppet last run on wtp1004 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [18:11:29] RECOVERY - puppet last run on wtp1011 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [18:11:29] RECOVERY - puppet last run on ssl1008 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [18:11:29] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [18:11:29] RECOVERY - puppet last run on elastic1011 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:11:29] RECOVERY - puppet last run on mw1163 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [18:11:30] RECOVERY - puppet last run on search1015 is OK: OK: Puppet is currently enabled, last run 1197 seconds ago with 0 failures [18:11:30] RECOVERY - puppet last run on amssq41 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [18:11:36] !log osmium - hhvm : Depends: libdouble-conversion1 but it is not going to be installed [18:11:39] RECOVERY - puppet last run on elastic1005 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [18:11:39] RECOVERY - puppet last run on db1001 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:11:39] RECOVERY - puppet last run on mw1023 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [18:11:39] RECOVERY - puppet last run on wtp1013 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [18:11:39] RECOVERY - puppet last run on search1023 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [18:11:41] Logged the message, Master [18:11:49] RECOVERY - puppet last run on db1070 is OK: OK: Puppet is currently enabled, last run 1201 seconds ago with 0 failures [18:11:49] RECOVERY - puppet last run on wtp1023 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [18:11:49] RECOVERY - puppet last run on amssq42 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [18:11:49] RECOVERY - puppet last run on mw1053 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [18:11:49] RECOVERY - puppet last run on mw1029 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [18:11:50] RECOVERY - puppet last run on mw1148 is OK: OK: Puppet is currently enabled, last run 1158 seconds ago with 0 failures [18:11:50] RECOVERY - puppet last run on rcs1001 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:11:51] RECOVERY - puppet last run on titanium is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [18:11:59] RECOVERY - puppet last run on mw1074 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:11:59] RECOVERY - puppet last run on elastic1002 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:11:59] RECOVERY - puppet last run on mw1087 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [18:12:09] RECOVERY - puppet last run on rcs1002 is OK: OK: Puppet is currently enabled, last run 1249 seconds ago with 0 failures [18:12:09] RECOVERY - puppet last run on amssq62 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [18:12:09] RECOVERY - puppet last run on lvs1001 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:12:09] RECOVERY - puppet last run on search1013 is OK: OK: Puppet is currently enabled, last run 1165 seconds ago with 0 failures [18:12:19] RECOVERY - puppet last run on rdb1002 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [18:12:19] RECOVERY - puppet last run on cp1060 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:12:29] RECOVERY - puppet last run on db1063 is OK: OK: Puppet is currently enabled, last run 1174 seconds ago with 0 failures [18:12:29] RECOVERY - puppet last run on nitrogen is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [18:12:29] RECOVERY - puppet last run on mc1007 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [18:12:29] RECOVERY - puppet last run on amssq38 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [18:12:39] RECOVERY - puppet last run on amslvs3 is OK: OK: Puppet is currently enabled, last run 1264 seconds ago with 0 failures [18:12:39] RECOVERY - puppet last run on wtp1007 is OK: OK: Puppet is currently enabled, last run 1240 seconds ago with 0 failures [18:12:39] RECOVERY - puppet last run on db1064 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [18:12:39] RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 1182 seconds ago with 0 failures [18:12:39] RECOVERY - puppet last run on cp3009 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [18:12:40] RECOVERY - puppet last run on rdb1003 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [18:12:40] RECOVERY - puppet last run on mw1121 is OK: OK: Puppet is currently enabled, last run 1186 seconds ago with 0 failures [18:12:49] RECOVERY - puppet last run on mw1167 is OK: OK: Puppet is currently enabled, last run 1168 seconds ago with 0 failures [18:12:49] RECOVERY - puppet last run on mw1105 is OK: OK: Puppet is currently enabled, last run 1245 seconds ago with 0 failures [18:12:49] RECOVERY - puppet last run on es1002 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [18:12:49] RECOVERY - puppet last run on ssl1007 is OK: OK: Puppet is currently enabled, last run 1199 seconds ago with 0 failures [18:12:49] RECOVERY - puppet last run on db1044 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [18:12:50] RECOVERY - puppet last run on analytics1011 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:12:50] RECOVERY - puppet last run on cp3012 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:12:51] RECOVERY - puppet last run on pc1003 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:12:54] ori: hhvm on osmium, unmet package dependencies.. not sure if you really want "apt-get -f install" [18:12:59] RECOVERY - puppet last run on db1061 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:12:59] RECOVERY - puppet last run on db1030 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:12:59] RECOVERY - puppet last run on mw1024 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [18:13:09] RECOVERY - puppet last run on wtp1003 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:13:09] RECOVERY - puppet last run on dbstore1002 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [18:13:09] RECOVERY - puppet last run on mw1077 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:13:09] RECOVERY - puppet last run on mw1108 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [18:13:09] RECOVERY - puppet last run on es1010 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:13:19] RECOVERY - puppet last run on ssl1001 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [18:13:19] RECOVERY - puppet last run on mw1033 is OK: OK: Puppet is currently enabled, last run 1194 seconds ago with 0 failures [18:13:19] RECOVERY - puppet last run on mw1201 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [18:13:19] RECOVERY - puppet last run on mercury is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:13:19] RECOVERY - puppet last run on analytics1014 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [18:13:20] does anyone see an issue here: http://pastie.org/9331819 [18:13:29] RECOVERY - puppet last run on analytics1027 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:13:29] RECOVERY - puppet last run on analytics1004 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:13:29] RECOVERY - puppet last run on db1011 is OK: OK: Puppet is currently enabled, last run 1197 seconds ago with 0 failures [18:13:29] RECOVERY - puppet last run on mw1022 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [18:13:29] RECOVERY - puppet last run on ms-fe3001 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:13:30] RECOVERY - puppet last run on mw1043 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [18:13:30] RECOVERY - puppet last run on mw1185 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:13:31] RECOVERY - puppet last run on mw1010 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [18:13:34] ACKNOWLEDGEMENT - DPKG on osmium is CRITICAL: DPKG CRITICAL dpkg reports broken packages ori.livneh testing HHVM packages on designated host [18:13:39] RECOVERY - puppet last run on labsdb1004 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [18:13:39] RECOVERY - puppet last run on search1022 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:13:39] RECOVERY - puppet last run on fluorine is OK: OK: Puppet is currently enabled, last run 1149 seconds ago with 0 failures [18:13:39] RECOVERY - puppet last run on mw1001 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:13:39] RECOVERY - puppet last run on mw1186 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [18:13:40] RECOVERY - puppet last run on calcium is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [18:13:40] RECOVERY - puppet last run on db1006 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [18:13:41] RECOVERY - puppet last run on db1033 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:13:49] RECOVERY - puppet last run on db1038 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:13:49] RECOVERY - puppet last run on mw1122 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:13:49] RECOVERY - puppet last run on iodine is OK: OK: Puppet is currently enabled, last run 1232 seconds ago with 0 failures [18:13:49] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:13:49] RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [18:13:50] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [18:13:59] RECOVERY - puppet last run on mw1209 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:13:59] RECOVERY - puppet last run on osm-db1002 is OK: OK: Puppet is currently enabled, last run 1154 seconds ago with 0 failures [18:13:59] RECOVERY - puppet last run on sanger is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [18:13:59] RECOVERY - puppet last run on cp1052 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [18:14:09] RECOVERY - puppet last run on search1006 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:14:09] RECOVERY - puppet last run on mw1139 is OK: OK: Puppet is currently enabled, last run 1225 seconds ago with 0 failures [18:14:09] RECOVERY - puppet last run on ms1002 is OK: OK: Puppet is currently enabled, last run 1224 seconds ago with 0 failures [18:14:09] RECOVERY - puppet last run on db1047 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:14:19] RECOVERY - puppet last run on ssl1006 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:14:19] RECOVERY - puppet last run on strontium is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [18:14:19] RECOVERY - puppet last run on lvs3003 is OK: OK: Puppet is currently enabled, last run 1171 seconds ago with 0 failures [18:14:19] RECOVERY - puppet last run on copper is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:14:19] RECOVERY - puppet last run on mw1093 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [18:14:29] RECOVERY - puppet last run on mw1016 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [18:14:29] RECOVERY - puppet last run on ms-be1011 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [18:14:29] RECOVERY - puppet last run on db60 is OK: OK: Puppet is currently enabled, last run 1159 seconds ago with 0 failures [18:14:29] RECOVERY - puppet last run on db1072 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:14:29] RECOVERY - puppet last run on mw1142 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [18:14:30] RECOVERY - puppet last run on mw1152 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:14:39] RECOVERY - puppet last run on cp1037 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [18:14:39] RECOVERY - puppet last run on stat1001 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [18:14:39] RECOVERY - puppet last run on search1011 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:14:49] RECOVERY - puppet last run on mw1219 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [18:14:49] RECOVERY - puppet last run on mw1091 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [18:14:49] RECOVERY - puppet last run on mw1112 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:14:49] RECOVERY - puppet last run on mw1090 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [18:14:49] RECOVERY - puppet last run on mw1204 is OK: OK: Puppet is currently enabled, last run 1183 seconds ago with 0 failures [18:14:50] RECOVERY - puppet last run on cp4020 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [18:14:56] <^d> paravoid: Everything ok? [18:14:59] RECOVERY - puppet last run on mw1107 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [18:14:59] RECOVERY - puppet last run on mw1193 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [18:14:59] RECOVERY - DPKG on osmium is OK: All packages OK [18:14:59] RECOVERY - puppet last run on amssq33 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [18:15:09] RECOVERY - puppet last run on mw1086 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [18:15:09] RECOVERY - puppet last run on ms-be1015 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:15:09] RECOVERY - puppet last run on cp1040 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [18:15:09] RECOVERY - puppet last run on elastic1003 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [18:15:09] RECOVERY - puppet last run on mw1215 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [18:15:10] RECOVERY - puppet last run on db71 is OK: OK: Puppet is currently enabled, last run 1158 seconds ago with 0 failures [18:15:10] PROBLEM - Unmerged changes on repository puppet on virt0 is CRITICAL: Fetching origin [18:15:11] RECOVERY - puppet last run on analytics1018 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [18:15:11] RECOVERY - puppet last run on mw1220 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [18:15:12] RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:15:19] RECOVERY - puppet last run on wtp1010 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:15:19] RECOVERY - puppet last run on linne is OK: OK: Puppet is currently enabled, last run 1173 seconds ago with 0 failures [18:15:19] RECOVERY - puppet last run on amssq31 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [18:15:29] RECOVERY - puppet last run on ssl3001 is OK: OK: Puppet is currently enabled, last run 1171 seconds ago with 0 failures [18:15:29] RECOVERY - puppet last run on cp1044 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [18:15:29] RECOVERY - puppet last run on search1012 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:15:29] RECOVERY - puppet last run on ms-fe1002 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [18:15:29] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 1179 seconds ago with 0 failures [18:15:30] RECOVERY - puppet last run on mw1027 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [18:15:39] RECOVERY - puppet last run on lvs4004 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [18:15:39] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [18:15:39] RECOVERY - puppet last run on virt1008 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [18:15:39] RECOVERY - puppet last run on cp4009 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [18:15:39] RECOVERY - puppet last run on amssq39 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [18:15:40] RECOVERY - puppet last run on mw1021 is OK: OK: Puppet is currently enabled, last run 1183 seconds ago with 0 failures [18:15:40] RECOVERY - puppet last run on db1037 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [18:15:41] RECOVERY - puppet last run on cp1067 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [18:15:41] RECOVERY - puppet last run on mw1064 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:15:42] RECOVERY - puppet last run on search1004 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [18:15:49] RECOVERY - puppet last run on mw1154 is OK: OK: Puppet is currently enabled, last run 1156 seconds ago with 0 failures [18:15:49] RECOVERY - puppet last run on analytics1003 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [18:15:49] RECOVERY - puppet last run on osm-cp1001 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:15:49] RECOVERY - puppet last run on mw1071 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [18:15:49] RECOVERY - puppet last run on radon is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [18:15:50] RECOVERY - puppet last run on wtp1008 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:15:50] RECOVERY - puppet last run on ms-be1002 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [18:15:51] RECOVERY - puppet last run on lvs1004 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [18:15:59] RECOVERY - puppet last run on wtp1001 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [18:15:59] RECOVERY - puppet last run on mw1066 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [18:15:59] RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 1243 seconds ago with 0 failures [18:15:59] RECOVERY - puppet last run on cp1054 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [18:16:09] RECOVERY - puppet last run on cp1053 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [18:16:09] RECOVERY - puppet last run on mw1143 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [18:16:09] RECOVERY - puppet last run on db1035 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [18:16:09] RECOVERY - puppet last run on mw1155 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:16:10] RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 1216 seconds ago with 0 failures [18:16:10] RECOVERY - puppet last run on cp4002 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [18:16:19] RECOVERY - puppet last run on mexia is OK: OK: Puppet is currently enabled, last run 1227 seconds ago with 0 failures [18:16:29] RECOVERY - puppet last run on db1056 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [18:16:29] RECOVERY - puppet last run on mw1037 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [18:16:29] RECOVERY - puppet last run on mw1207 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [18:16:29] RECOVERY - puppet last run on mw1135 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [18:16:29] RECOVERY - puppet last run on cp4012 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:16:30] RECOVERY - puppet last run on search1003 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [18:16:30] RECOVERY - puppet last run on mw1104 is OK: OK: Puppet is currently enabled, last run 1200 seconds ago with 0 failures [18:16:31] RECOVERY - puppet last run on cp1045 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [18:16:31] RECOVERY - puppet last run on mw1131 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:16:39] RECOVERY - puppet last run on lvs1003 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:16:39] RECOVERY - puppet last run on eeden is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [18:16:39] RECOVERY - puppet last run on bast1001 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [18:16:39] RECOVERY - puppet last run on mw1113 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [18:16:39] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:16:49] RECOVERY - puppet last run on analytics1021 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [18:16:49] RECOVERY - puppet last run on fenari is OK: OK: Puppet is currently enabled, last run 1134 seconds ago with 1 failures [18:16:49] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [18:16:49] RECOVERY - puppet last run on cp3018 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:16:49] RECOVERY - puppet last run on analytics1012 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [18:16:50] RECOVERY - puppet last run on amssq37 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [18:16:50] RECOVERY - puppet last run on mc1015 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [18:16:51] RECOVERY - puppet last run on cp1068 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [18:16:51] RECOVERY - puppet last run on elastic1009 is OK: OK: Puppet is currently enabled, last run 1205 seconds ago with 0 failures [18:16:52] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [18:16:52] RECOVERY - puppet last run on ssl1003 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [18:16:59] RECOVERY - puppet last run on mw1047 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [18:17:09] RECOVERY - puppet last run on tmh1001 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:17:09] RECOVERY - puppet last run on mw1194 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [18:17:09] RECOVERY - puppet last run on cp3021 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:17:09] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [18:17:29] RECOVERY - puppet last run on cp1066 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:17:29] RECOVERY - puppet last run on es1004 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [18:17:29] RECOVERY - puppet last run on mw1103 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:17:29] RECOVERY - puppet last run on cp3011 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [18:17:29] RECOVERY - puppet last run on lanthanum is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [18:17:30] RECOVERY - puppet last run on cp1064 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [18:17:30] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 1157 seconds ago with 0 failures [18:17:31] RECOVERY - puppet last run on hydrogen is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [18:17:31] RECOVERY - puppet last run on zinc is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [18:17:32] RECOVERY - puppet last run on analytics1009 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [18:17:32] RECOVERY - puppet last run on palladium is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [18:17:33] RECOVERY - puppet last run on db1005 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [18:17:39] RECOVERY - puppet last run on amssq50 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:17:39] RECOVERY - puppet last run on ms-be1014 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:17:39] RECOVERY - puppet last run on labsdb1005 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [18:17:39] RECOVERY - puppet last run on elastic1013 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [18:17:48] (03Restored) 10Scottlee: Fixed spacing and lint rules for manifests/misc files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142483 (owner: 10Scottlee) [18:17:49] RECOVERY - puppet last run on db1009 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:17:49] RECOVERY - puppet last run on mw1019 is OK: OK: Puppet is currently enabled, last run 1184 seconds ago with 0 failures [18:17:49] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 1184 seconds ago with 0 failures [18:17:49] RECOVERY - puppet last run on db1049 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [18:17:49] RECOVERY - puppet last run on mw1058 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [18:17:50] RECOVERY - puppet last run on db1068 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [18:17:50] RECOVERY - puppet last run on wtp1014 is OK: OK: Puppet is currently enabled, last run 1162 seconds ago with 0 failures [18:17:51] RECOVERY - puppet last run on mw1128 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [18:17:51] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [18:17:52] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [18:17:59] RECOVERY - puppet last run on mw1078 is OK: OK: Puppet is currently enabled, last run 1194 seconds ago with 0 failures [18:17:59] RECOVERY - puppet last run on ms-be3004 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:17:59] RECOVERY - puppet last run on mw1199 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [18:18:09] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [18:18:09] RECOVERY - puppet last run on search1019 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [18:18:09] RECOVERY - puppet last run on virt1005 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:18:09] RECOVERY - puppet last run on mw1020 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:18:09] RECOVERY - puppet last run on elastic1010 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [18:18:10] RECOVERY - puppet last run on cp4011 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [18:18:10] RECOVERY - puppet last run on mw1015 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [18:18:11] RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [18:18:29] RECOVERY - puppet last run on search1008 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [18:18:29] RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:18:29] RECOVERY - puppet last run on mw1101 is OK: OK: Puppet is currently enabled, last run 1229 seconds ago with 0 failures [18:18:29] RECOVERY - puppet last run on search1009 is OK: OK: Puppet is currently enabled, last run 1152 seconds ago with 0 failures [18:18:29] RECOVERY - puppet last run on mw1070 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [18:18:30] RECOVERY - puppet last run on mw1191 is OK: OK: Puppet is currently enabled, last run 1145 seconds ago with 0 failures [18:18:30] RECOVERY - puppet last run on search1021 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [18:18:31] RECOVERY - puppet last run on mw1083 is OK: OK: Puppet is currently enabled, last run 1201 seconds ago with 0 failures [18:18:31] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 1211 seconds ago with 0 failures [18:18:39] RECOVERY - puppet last run on mw1182 is OK: OK: Puppet is currently enabled, last run 1154 seconds ago with 0 failures [18:18:39] RECOVERY - puppet last run on mw1075 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:18:39] RECOVERY - puppet last run on tantalum is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [18:18:39] RECOVERY - puppet last run on cp3019 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [18:18:39] RECOVERY - puppet last run on mw1179 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:18:40] RECOVERY - puppet last run on lvs4001 is OK: OK: Puppet is currently enabled, last run 1165 seconds ago with 0 failures [18:18:49] RECOVERY - puppet last run on mw1095 is OK: OK: Puppet is currently enabled, last run 1242 seconds ago with 0 failures [18:18:49] RECOVERY - puppet last run on es1005 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [18:18:49] RECOVERY - puppet last run on mw1127 is OK: OK: Puppet is currently enabled, last run 1157 seconds ago with 0 failures [18:18:49] RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [18:18:49] RECOVERY - puppet last run on mw1214 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [18:18:50] RECOVERY - puppet last run on nickel is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [18:18:50] RECOVERY - puppet last run on tmh1002 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [18:18:59] RECOVERY - puppet last run on labsdb1002 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [18:19:09] RECOVERY - puppet last run on mw1169 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:19:09] RECOVERY - puppet last run on analytics1024 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:19:09] RECOVERY - puppet last run on cp3007 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [18:19:09] RECOVERY - puppet last run on amssq52 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [18:19:09] RECOVERY - puppet last run on db1041 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [18:19:10] RECOVERY - puppet last run on es1003 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:19:10] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [18:19:11] RECOVERY - puppet last run on cp1065 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:19:11] RECOVERY - puppet last run on elastic1016 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [18:19:12] RECOVERY - puppet last run on es4 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:19:19] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [18:19:29] RECOVERY - puppet last run on ms-be1005 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [18:19:29] RECOVERY - puppet last run on erbium is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 1 failures [18:19:29] RECOVERY - puppet last run on mw1094 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [18:19:38] (03PS1) 10Dzahn: put legalpad phab behind misc. varnish [operations/puppet] - 10https://gerrit.wikimedia.org/r/142601 [18:19:39] RECOVERY - puppet last run on virt0 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:19:39] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [18:19:39] RECOVERY - puppet last run on mc1008 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [18:19:39] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [18:19:39] RECOVERY - puppet last run on mw1136 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:19:58] RECOVERY - puppet last run on mw1036 is OK: OK: Puppet is currently enabled, last run 1201 seconds ago with 0 failures [18:19:58] RECOVERY - puppet last run on es10 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [18:19:58] RECOVERY - puppet last run on cp4013 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [18:19:58] RECOVERY - puppet last run on amssq45 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [18:19:58] RECOVERY - puppet last run on ssl1004 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:19:59] RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 1185 seconds ago with 0 failures [18:19:59] RECOVERY - puppet last run on virt1009 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [18:19:59] (03PS3) 10Scottlee: Fixed spacing and lint rules for manifests/misc files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142483 [18:20:00] RECOVERY - puppet last run on cp1069 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [18:20:00] RECOVERY - puppet last run on mw1096 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [18:20:01] RECOVERY - puppet last run on amssq57 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:20:01] RECOVERY - puppet last run on cp1057 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [18:20:02] RECOVERY - puppet last run on cp1051 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [18:20:02] RECOVERY - puppet last run on cp4017 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [18:20:03] RECOVERY - puppet last run on amslvs4 is OK: OK: Puppet is currently enabled, last run 1210 seconds ago with 0 failures [18:20:03] RECOVERY - puppet last run on analytics1019 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [18:20:04] RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [18:20:04] RECOVERY - puppet last run on search1014 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [18:20:05] RECOVERY - puppet last run on mw1013 is OK: OK: Puppet is currently enabled, last run 1203 seconds ago with 0 failures [18:20:05] RECOVERY - puppet last run on mw1184 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:20:06] RECOVERY - puppet last run on mw1138 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [18:20:08] RECOVERY - puppet last run on dataset2 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 1 failures [18:20:08] RECOVERY - puppet last run on neon is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [18:20:08] RECOVERY - puppet last run on es1006 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:20:18] RECOVERY - puppet last run on zirconium is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [18:20:18] RECOVERY - puppet last run on db1007 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [18:20:18] RECOVERY - puppet last run on mw1109 is OK: OK: Puppet is currently enabled, last run 1148 seconds ago with 0 failures [18:20:28] RECOVERY - puppet last run on mw1040 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [18:20:28] RECOVERY - puppet last run on mw1196 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [18:20:28] RECOVERY - puppet last run on mw1192 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [18:20:38] RECOVERY - puppet last run on amssq58 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [18:20:38] RECOVERY - puppet last run on mc1010 is OK: OK: Puppet is currently enabled, last run 1213 seconds ago with 0 failures [18:20:38] RECOVERY - puppet last run on tarin is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [18:20:38] RECOVERY - puppet last run on mw1035 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [18:20:39] RECOVERY - puppet last run on es7 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [18:20:39] RECOVERY - puppet last run on mw1062 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [18:20:48] RECOVERY - puppet last run on es1009 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [18:20:48] RECOVERY - puppet last run on mw1147 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [18:20:58] RECOVERY - puppet last run on mc1011 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:20:58] RECOVERY - puppet last run on mw1115 is OK: OK: Puppet is currently enabled, last run 1148 seconds ago with 0 failures [18:21:08] RECOVERY - puppet last run on labsdb1001 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:21:08] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [18:21:08] RECOVERY - puppet last run on cp1043 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:21:08] RECOVERY - puppet last run on mw1124 is OK: OK: Puppet is currently enabled, last run 1221 seconds ago with 0 failures [18:21:18] RECOVERY - puppet last run on cp3013 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:21:18] RECOVERY - puppet last run on mw1072 is OK: OK: Puppet is currently enabled, last run 1111 seconds ago with 0 failures [18:21:18] RECOVERY - puppet last run on ytterbium is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [18:21:18] RECOVERY - puppet last run on wtp1024 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [18:21:28] RECOVERY - puppet last run on mw1161 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [18:21:28] RECOVERY - puppet last run on search1020 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [18:21:28] RECOVERY - puppet last run on ms1001 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:21:28] RECOVERY - puppet last run on virt1002 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:21:28] RECOVERY - puppet last run on ssl3003 is OK: OK: Puppet is currently enabled, last run 1200 seconds ago with 0 failures [18:21:38] RECOVERY - puppet last run on mw1132 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [18:21:38] RECOVERY - puppet last run on ms-fe1003 is OK: OK: Puppet is currently enabled, last run 1222 seconds ago with 0 failures [18:21:38] RECOVERY - puppet last run on mw1028 is OK: OK: Puppet is currently enabled, last run 1182 seconds ago with 0 failures [18:21:38] RECOVERY - puppet last run on mw1080 is OK: OK: Puppet is currently enabled, last run 1126 seconds ago with 0 failures [18:21:38] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:21:39] RECOVERY - puppet last run on vanadium is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [18:21:39] RECOVERY - puppet last run on cp3022 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [18:21:40] RECOVERY - puppet last run on logstash1003 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:21:40] RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [18:21:41] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:21:41] RECOVERY - puppet last run on mw1130 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [18:21:48] RECOVERY - puppet last run on db1010 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [18:21:48] RECOVERY - puppet last run on mw1089 is OK: OK: Puppet is currently enabled, last run 1200 seconds ago with 0 failures [18:21:48] RECOVERY - puppet last run on mc1009 is OK: OK: Puppet is currently enabled, last run 1221 seconds ago with 0 failures [18:21:48] RECOVERY - puppet last run on ms-be1013 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [18:21:48] RECOVERY - puppet last run on cp4007 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [18:21:58] RECOVERY - puppet last run on dbstore1001 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [18:22:08] RECOVERY - puppet last run on mc1004 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [18:22:08] RECOVERY - puppet last run on mw1048 is OK: OK: Puppet is currently enabled, last run 1153 seconds ago with 0 failures [18:22:08] RECOVERY - puppet last run on wtp1017 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [18:22:08] RECOVERY - puppet last run on mw1038 is OK: OK: Puppet is currently enabled, last run 1163 seconds ago with 0 failures [18:22:09] RECOVERY - puppet last run on mw1134 is OK: OK: Puppet is currently enabled, last run 1222 seconds ago with 0 failures [18:22:09] RECOVERY - puppet last run on wtp1009 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [18:22:18] RECOVERY - puppet last run on mw1031 is OK: OK: Puppet is currently enabled, last run 1162 seconds ago with 0 failures [18:22:28] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 1161 seconds ago with 0 failures [18:22:28] RECOVERY - puppet last run on mw1145 is OK: OK: Puppet is currently enabled, last run 1161 seconds ago with 0 failures [18:22:28] RECOVERY - puppet last run on rdb1004 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:22:28] RECOVERY - puppet last run on aluminium is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [18:22:28] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:22:29] RECOVERY - puppet last run on logstash1001 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:22:29] RECOVERY - puppet last run on amssq44 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [18:22:30] RECOVERY - puppet last run on wtp1019 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [18:22:38] RECOVERY - puppet last run on cp1059 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [18:22:38] RECOVERY - puppet last run on mw1178 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [18:22:38] RECOVERY - puppet last run on pc1001 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:22:38] RECOVERY - puppet last run on mc1016 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:22:38] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 1174 seconds ago with 0 failures [18:22:48] RECOVERY - puppet last run on mw1106 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:22:48] RECOVERY - puppet last run on mw1059 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [18:22:48] RECOVERY - puppet last run on lvs4002 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:22:48] RECOVERY - puppet last run on cp3015 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [18:22:58] RECOVERY - puppet last run on analytics1017 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [18:22:58] RECOVERY - puppet last run on mw1140 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:23:08] RECOVERY - puppet last run on mw1005 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [18:23:08] RECOVERY - puppet last run on cp1070 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [18:23:08] RECOVERY - puppet last run on mw1007 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [18:23:08] RECOVERY - puppet last run on mw1067 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [18:23:08] RECOVERY - puppet last run on mw1200 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [18:23:09] RECOVERY - puppet last run on amslvs2 is OK: OK: Puppet is currently enabled, last run 1195 seconds ago with 0 failures [18:23:09] RECOVERY - puppet last run on amssq43 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [18:23:10] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:23:10] RECOVERY - puppet last run on db1017 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [18:23:11] RECOVERY - puppet last run on analytics1025 is OK: OK: Puppet is currently enabled, last run 1219 seconds ago with 0 failures [18:23:11] RECOVERY - puppet last run on db1029 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [18:23:12] RECOVERY - puppet last run on ms-be1004 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:23:12] RECOVERY - puppet last run on ssl1002 is OK: OK: Puppet is currently enabled, last run 1211 seconds ago with 0 failures [18:23:13] RECOVERY - puppet last run on searchidx1001 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [18:23:18] RECOVERY - puppet last run on magnesium is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:23:18] RECOVERY - puppet last run on amssq59 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [18:23:18] RECOVERY - puppet last run on es1001 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [18:23:18] RECOVERY - puppet last run on mw1012 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [18:23:18] RECOVERY - puppet last run on db73 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:23:28] RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [18:23:28] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [18:23:28] RECOVERY - puppet last run on mw1045 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:23:28] RECOVERY - puppet last run on caesium is OK: OK: Puppet is currently enabled, last run 1225 seconds ago with 0 failures [18:23:28] RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:23:38] RECOVERY - puppet last run on mw1187 is OK: OK: Puppet is currently enabled, last run 1139 seconds ago with 0 failures [18:23:38] RECOVERY - puppet last run on mw1041 is OK: OK: Puppet is currently enabled, last run 1227 seconds ago with 0 failures [18:23:38] RECOVERY - puppet last run on mw1063 is OK: OK: Puppet is currently enabled, last run 1226 seconds ago with 0 failures [18:23:38] RECOVERY - puppet last run on snapshot1003 is OK: OK: Puppet is currently enabled, last run 1149 seconds ago with 0 failures [18:23:38] RECOVERY - puppet last run on amssq54 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:23:39] RECOVERY - puppet last run on carbon is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [18:23:39] RECOVERY - puppet last run on db1065 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:23:46] aww icinga-wm, is just happy everything is ok [18:23:48] RECOVERY - puppet last run on francium is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [18:23:48] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:23:48] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [18:23:58] RECOVERY - puppet last run on mw1006 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [18:23:58] RECOVERY - puppet last run on mw1026 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [18:24:08] RECOVERY - puppet last run on cerium is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:24:08] RECOVERY - puppet last run on mw1082 is OK: OK: Puppet is currently enabled, last run 1170 seconds ago with 0 failures [18:24:08] RECOVERY - puppet last run on db74 is OK: OK: Puppet is currently enabled, last run 1178 seconds ago with 0 failures [18:24:08] RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:24:08] RECOVERY - puppet last run on elastic1004 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:24:09] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 1159 seconds ago with 0 failures [18:24:18] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [18:24:18] RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [18:24:28] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [18:24:28] RECOVERY - puppet last run on wtp1006 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [18:24:28] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [18:24:28] RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:24:28] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 1107 seconds ago with 0 failures [18:24:29] RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:24:29] RECOVERY - puppet last run on analytics1020 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [18:24:30] RECOVERY - puppet last run on elastic1001 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [18:24:30] RECOVERY - puppet last run on cp1055 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [18:24:31] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:24:31] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:24:38] RECOVERY - puppet last run on elastic1007 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [18:24:38] RECOVERY - puppet last run on helium is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [18:24:38] RECOVERY - puppet last run on wtp1020 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [18:24:38] RECOVERY - puppet last run on search1010 is OK: OK: Puppet is currently enabled, last run 1196 seconds ago with 0 failures [18:24:38] RECOVERY - puppet last run on tridge is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [18:24:39] RECOVERY - puppet last run on cp4006 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [18:24:39] RECOVERY - puppet last run on elastic1008 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [18:24:40] RECOVERY - puppet last run on mc1002 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [18:24:40] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:24:41] RECOVERY - puppet last run on mw1160 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [18:24:41] RECOVERY - puppet last run on potassium is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [18:24:42] RECOVERY - puppet last run on es1008 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [18:24:42] RECOVERY - puppet last run on search1016 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [18:24:48] RECOVERY - puppet last run on ms-be1003 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [18:24:48] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [18:24:48] RECOVERY - puppet last run on mw1153 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:24:48] RECOVERY - puppet last run on amssq61 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [18:24:48] RECOVERY - puppet last run on cp1039 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [18:24:58] RECOVERY - puppet last run on db1066 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [18:24:59] YuviPanda: icinga's just all "hey guys hey guys hey guys! Look! Recovery!" [18:25:02] greg-g: :) [18:25:08] RECOVERY - puppet last run on amssq53 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [18:25:08] RECOVERY - puppet last run on lead is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:25:08] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [18:25:09] RECOVERY - puppet last run on db1022 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:25:09] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 1172 seconds ago with 0 failures [18:25:09] RECOVERY - puppet last run on ms1004 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [18:25:09] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [18:25:10] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [18:25:10] RECOVERY - puppet last run on mw1176 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [18:25:18] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 1233 seconds ago with 0 failures [18:25:18] RECOVERY - puppet last run on mw1205 is OK: OK: Puppet is currently enabled, last run 1225 seconds ago with 0 failures [18:25:28] RECOVERY - puppet last run on mw1217 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [18:25:28] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 1240 seconds ago with 0 failures [18:25:28] RECOVERY - puppet last run on elastic1018 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [18:25:28] RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [18:25:28] RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [18:25:29] RECOVERY - puppet last run on mw1099 is OK: OK: Puppet is currently enabled, last run 1251 seconds ago with 0 failures [18:25:29] RECOVERY - puppet last run on amssq32 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [18:25:38] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 1193 seconds ago with 0 failures [18:25:38] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 1264 seconds ago with 0 failures [18:25:48] RECOVERY - puppet last run on ruthenium is OK: OK: Puppet is currently enabled, last run 1189 seconds ago with 0 failures [18:25:48] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 1251 seconds ago with 0 failures [18:25:48] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [18:25:55] (03PS2) 10Chad: Remove "Cirrus as alternative" switches [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/140756 [18:25:57] (03PS2) 10Chad: Pool 2 wikis (dewiki, frwiki, jawiki) get Cirrus as primary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/140754 [18:25:58] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [18:25:58] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [18:25:58] RECOVERY - puppet last run on mc1003 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:25:58] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:25:58] RECOVERY - puppet last run on virt1006 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:25:59] (03PS2) 10Chad: enwiki gets Cirrus as primary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/140755 [18:26:01] (03PS2) 10Chad: Move commons over to Cirrus as primary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/140752 [18:26:03] (03PS2) 10Chad: Move remaining pool 3 wikis to Cirrus as primary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/140753 [18:26:05] (03PS1) 10Chad: Move remaining pool 4 lsearchd wikis (except commons) to Cirrus [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142606 [18:26:07] (03PS1) 10Chad: Reverse Cirrus config, all wikis get it by default now [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142607 [18:26:08] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [18:26:08] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:26:08] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [18:26:08] RECOVERY - puppet last run on search1001 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [18:26:08] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [18:26:09] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [18:26:09] RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:26:10] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 1 failures [18:26:10] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 1121 seconds ago with 0 failures [18:26:11] RECOVERY - puppet last run on wtp1016 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [18:26:18] RECOVERY - puppet last run on holmium is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:26:18] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [18:26:18] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:26:18] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [18:26:18] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [18:26:28] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [18:26:28] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [18:26:28] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [18:26:28] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [18:26:28] RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 1144 seconds ago with 0 failures [18:26:29] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [18:26:38] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:26:38] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:26:38] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [18:26:38] RECOVERY - puppet last run on mw1011 is OK: OK: Puppet is currently enabled, last run 1153 seconds ago with 0 failures [18:26:38] RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 1147 seconds ago with 0 failures [18:26:39] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [18:26:39] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [18:26:40] RECOVERY - puppet last run on logstash1002 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [18:26:40] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [18:26:41] RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:26:48] RECOVERY - puppet last run on mw1177 is OK: OK: Puppet is currently enabled, last run 1220 seconds ago with 0 failures [18:26:48] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [18:26:58] RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 1230 seconds ago with 0 failures [18:26:58] RECOVERY - puppet last run on db1028 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:26:58] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:27:08] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [18:27:08] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [18:27:08] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 1184 seconds ago with 0 failures [18:27:09] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [18:27:09] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [18:27:18] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [18:27:18] RECOVERY - puppet last run on mw1129 is OK: OK: Puppet is currently enabled, last run 1181 seconds ago with 0 failures [18:27:18] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 1201 seconds ago with 0 failures [18:27:28] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [18:27:28] RECOVERY - puppet last run on mw1002 is OK: OK: Puppet is currently enabled, last run 1199 seconds ago with 0 failures [18:27:28] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [18:27:32] I need a merge on this patch: https://gerrit.wikimedia.org/r/#/c/137804/ [18:27:38] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 1214 seconds ago with 0 failures [18:27:38] RECOVERY - puppet last run on mw1126 is OK: OK: Puppet is currently enabled, last run 1214 seconds ago with 0 failures [18:27:38] RECOVERY - puppet last run on db1023 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [18:27:39] RECOVERY - puppet last run on amssq46 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:27:39] RECOVERY - puppet last run on analytics1010 is OK: OK: Puppet is currently enabled, last run 1220 seconds ago with 0 failures [18:27:39] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [18:27:42] Don't make me start pinging individuals! :p [18:27:48] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [18:27:48] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [18:27:58] RECOVERY - puppet last run on search1007 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [18:27:58] RECOVERY - puppet last run on cp1058 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [18:27:58] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [18:27:58] RECOVERY - puppet last run on ms-be3003 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [18:28:18] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [18:28:18] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [18:28:18] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [18:28:28] RECOVERY - puppet last run on amssq60 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [18:28:38] RECOVERY - puppet last run on amssq48 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [18:29:38] RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [18:29:43] (03Restored) 10Scottlee: Fixed spacing and lint rules for manifests files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142479 (owner: 10Scottlee) [18:32:08] I assume the spam is just the first output from the check finally working everywhere? [18:34:42] nope, it's the new verbosity setting [18:34:47] it's going to be like this from now on [18:34:57] !log updated puppet repo on virt0 [18:35:03] Logged the message, Master [18:35:08] RECOVERY - Unmerged changes on repository puppet on virt0 is OK: Fetching origin [18:35:33] awight: you don't actually need a merge on that. you can merge it yourself, but you should wait for a deployment window [18:35:44] ori: I don't have +2 here though [18:35:45] awight: or simply add it to the SWAT window on monday morning [18:35:58] sure you do [18:36:03] it's mediawiki-config, not puppet [18:36:14] if you need it out urgently, check with greg-g [18:36:16] * awight looks again [18:36:25] linky? [18:36:31] greg-g: https://gerrit.wikimedia.org/r/#/c/137804/ [18:36:55] what the... now I do have +2. Anyway, I wrote it so I'm loathe to merge unilaterally. [18:37:06] (03PS1) 10Dzahn: switch radon from public to private IP [operations/dns] - 10https://gerrit.wikimedia.org/r/142611 [18:37:15] i didn't change your privs, i just noticed that the box next to your name wasn't greyed out [18:37:22] which it would have been if you had been unable to merge [18:37:38] RECOVERY - puppet last run on db1053 is OK: OK: Puppet is currently enabled, last run 915 seconds ago with 0 failures [18:37:38] RECOVERY - puppet last run on db1054 is OK: OK: Puppet is currently enabled, last run 351 seconds ago with 0 failures [18:37:43] I must have had a brain fart... [18:38:06] awight: monday morning swat please [18:38:08] RECOVERY - puppet last run on db1057 is OK: OK: Puppet is currently enabled, last run 421 seconds ago with 0 failures [18:38:12] the norm with mediawiki-config changes is that they are self-merged by the person deploying them just before the deployment [18:38:21] so just add it to swat for monday and it'll get taken care of for you [18:38:26] aha ok thanks [18:38:28] RECOVERY - puppet last run on db1058 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:38:58] (03CR) 10Dzahn: [C: 031] "11:34 < RobH> its basically best to assume its brand new from puppet standpoint and salt" [operations/dns] - 10https://gerrit.wikimedia.org/r/142611 (owner: 10Dzahn) [18:39:31] RECOVERY - puppet last run on db1045 is OK: OK: Puppet is currently enabled, last run 217 seconds ago with 0 failures [18:39:42] (03PS3) 10Scottlee: Fixed spacing and lint rules for manifests files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142479 [18:39:45] (03CR) 10RobH: "don't forget to remove the old salt and puppet keys (as well as clear the old entry from puppetdb)" [operations/dns] - 10https://gerrit.wikimedia.org/r/142611 (owner: 10Dzahn) [18:40:46] (03PS1) 10Ottomata: Use mapreduce_shuffle! This is correct [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142613 [18:40:55] (03CR) 10Ottomata: [C: 032 V: 032] Use mapreduce_shuffle! This is correct [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142613 (owner: 10Ottomata) [18:40:57] (03CR) 10jenkins-bot: [V: 04-1] Fixed spacing and lint rules for manifests files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142479 (owner: 10Scottlee) [18:41:24] (03PS2) 10Dzahn: switch radon from public to private IP [operations/dns] - 10https://gerrit.wikimedia.org/r/142611 [18:42:50] (03Abandoned) 10Dzahn: add varnish backend probe for legalpad [operations/puppet] - 10https://gerrit.wikimedia.org/r/142594 (owner: 10Dzahn) [18:48:54] (03CR) 10RobH: [C: 031] "This looks good, just putting +1 and leaving for owner to merge. Don't forget the associated dns cname change for the service url." [operations/puppet] - 10https://gerrit.wikimedia.org/r/142601 (owner: 10Dzahn) [18:50:17] (03CR) 10Aaron Schulz: [C: 032] Limit the number of expensive thumbnails processed at the same time [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/135704 (https://bugzilla.wikimedia.org/65691) (owner: 10Gergő Tisza) [18:52:13] (03CR) 10Dzahn: "thanks, yep, that DNS change is here: https://gerrit.wikimedia.org/r/#/c/142597/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/142601 (owner: 10Dzahn) [18:56:19] (03Merged) 10jenkins-bot: Limit the number of expensive thumbnails processed at the same time [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/135704 (https://bugzilla.wikimedia.org/65691) (owner: 10Gergő Tisza) [18:57:37] !log aaron Synchronized wmf-config/PoolCounterSettings-eqiad.php: Pre-set FileRenderExpensive config [18:57:41] Logged the message, Master [18:59:59] (03CR) 10Dzahn: [C: 031] Add grafana.wikimedia.org [operations/dns] - 10https://gerrit.wikimedia.org/r/133275 (owner: 10Ori.livneh) [19:01:20] (03CR) 10Dzahn: Make gordon an alternate to dickson (031 comment) [operations/dns] - 10https://gerrit.wikimedia.org/r/115093 (owner: 10coren) [19:03:24] mutante: I've never heard of 'tesla', was it decommed long ago? [19:03:44] (03CR) 10Dzahn: [C: 031] rm tesla (old virt box) [operations/dns] - 10https://gerrit.wikimedia.org/r/94115 (owner: 10Jeremyb) [19:03:56] andrewbogott: it's pre-labs virtualization [19:04:00] yes [19:04:17] i once removed most of it when RyanLane was still here [19:04:17] long time ago? [19:04:18] * Reedy grins [19:04:25] all i remember is that he said to keep mgmt [19:04:31] mutante: ok, then I have no opinion [19:04:37] but nowadays, that mgmt IP doesnt reply anymore anyways [19:04:48] and it's in the Tampa IP range [19:05:12] (03PS3) 10Ori.livneh: mediawiki_singlenode: port apache::vhost to apache::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/142206 [19:05:16] andrewbogott: ^ [19:05:57] (03CR) 10Ori.livneh: [C: 032] refreshDomainRedirects: fix off-by-one error [operations/apache-config] - 10https://gerrit.wikimedia.org/r/138297 (owner: 10Ori.livneh) [19:07:05] (03CR) 10Andrew Bogott: [C: 031] "Looks right. I'll set up a test case if you haven't already tried it on a live install..." [operations/puppet] - 10https://gerrit.wikimedia.org/r/142206 (owner: 10Ori.livneh) [19:07:44] (03PS3) 10Dzahn: rm tesla (old virt box) [operations/dns] - 10https://gerrit.wikimedia.org/r/94115 (owner: 10Jeremyb) [19:08:16] mutante: tesla mgmt is gone right? [19:08:19] cuz the server is long gone [19:08:21] we sold it [19:08:52] (03PS4) 10Dzahn: rm tesla (old virt box) [operations/dns] - 10https://gerrit.wikimedia.org/r/94115 (owner: 10Jeremyb) [19:09:03] yea, it is, nm [19:09:05] RobH: yea, it was an old change sitting in gerrit, trying to clean up [19:09:09] and ..after rebasing it [19:09:16] it turns into an even smaller change [19:09:18] its just removing the comments, heh [19:09:22] still needed [19:09:47] well, yea, that "usability" zone [19:09:50] but killing it :) [19:09:59] the change is still needed i mean [19:10:03] not the pointless comments [19:10:06] =] [19:10:20] (no more usability zonename to care about ) [19:10:29] (03PS3) 10Ori.livneh: role::deployment: port apache::vhost to apache::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/142205 [19:10:31] andrewbogott: thanks [19:10:39] (03CR) 10RobH: [C: 031] rm tesla (old virt box) [operations/dns] - 10https://gerrit.wikimedia.org/r/94115 (owner: 10Jeremyb) [19:10:41] (03PS5) 10Dzahn: rm tesla/usability zone [operations/dns] - 10https://gerrit.wikimedia.org/r/94115 (owner: 10Jeremyb) [19:10:48] (03CR) 10Dzahn: [C: 032] rm tesla/usability zone [operations/dns] - 10https://gerrit.wikimedia.org/r/94115 (owner: 10Jeremyb) [19:11:11] andrewbogott: re: > I'll set up a test case if you haven't already tried it on a live install... [19:11:18] andrewbogott: should i merge the patch for that? or explicitly hold off on merging? [19:11:23] (and no, i haven't yet..) [19:11:31] hold off, I'll test [19:11:46] (03CR) 10jenkins-bot: [V: 04-1] role::deployment: port apache::vhost to apache::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/142205 (owner: 10Ori.livneh) [19:11:52] andrewbogott: gotcha [19:12:05] RobH: hehe, that change was uploaded November 2013 [19:13:28] (03PS4) 10Ori.livneh: role::deployment: port apache::vhost to apache::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/142205 [19:14:56] (03PS3) 10Yuvipanda: nginx: Make nginx package name configurable [operations/puppet/nginx] - 10https://gerrit.wikimedia.org/r/142499 (https://bugzilla.wikimedia.org/67188) [19:14:58] ori: ^ updated to have 'variants' [19:14:59] (03CR) 10jenkins-bot: [V: 04-1] nginx: Make nginx package name configurable [operations/puppet/nginx] - 10https://gerrit.wikimedia.org/r/142499 (https://bugzilla.wikimedia.org/67188) (owner: 10Yuvipanda) [19:15:04] aww, syntax [19:16:07] (03PS4) 10Yuvipanda: nginx: Make nginx package name configurable [operations/puppet/nginx] - 10https://gerrit.wikimedia.org/r/142499 (https://bugzilla.wikimedia.org/67188) [19:16:10] um, shit -- ori, it looks like that class was already broken. So, lemme fix that first... [19:18:50] YuviPanda: looks good, one small suggestion [19:18:55] ori: sure [19:19:13] i'd also declare 'nginx-common'. all of the variants depend on it, but delcaring it explicitly means anyone using the module has a package name they know they can require => etc. [19:19:49] so i'd just change it to: [19:19:53] package { [ "nginx-${variant}", "nginx-${variant}-dbg", 'nginx-common' ]: } [19:19:56] ori: right. [19:20:04] ori: hmm, can't they just depend on the class itself? [19:20:14] yes [19:20:15] hmm, I don't remember if that works [19:20:17] it's not strictly required [19:20:21] ori: so why a package specifically? [19:20:22] but it's still nice [19:20:30] it feels like exposing an implementation details [19:20:31] *detail [19:20:32] (03CR) 10BryanDavis: [C: 031] "LGTM. I should be able to rebase I905a618 on this and get rid of my apache conf template there." [operations/puppet] - 10https://gerrit.wikimedia.org/r/142205 (owner: 10Ori.livneh) [19:20:46] YuviPanda: you might want to plop a file in /etc/nginx but do so before the service is started [19:20:51] if you require the class you get everything [19:21:05] if you do it before the class /etc/nginx won't be there [19:21:07] ori: ah, hmm. that sounds right. in fact I think I might have to do that in the dynamicproxy itself. [19:21:24] (03PS1) 10Dzahn: move Cyprus from Asia to Europe [operations/dns] - 10https://gerrit.wikimedia.org/r/142622 [19:21:26] (03CR) 10BryanDavis: [C: 04-1] "Should be rebased on I12c6a04 so that apache config template can be dropped." [operations/puppet] - 10https://gerrit.wikimedia.org/r/142407 (owner: 10BryanDavis) [19:21:59] ori, ever tried and had any success using custom module facts in vagrant? [19:22:17] ottomata: yes [19:22:25] i can't remember where now, let me grep mediawiki-vagrant [19:22:48] (03PS5) 10Yuvipanda: nginx: Make nginx package name configurable [operations/puppet/nginx] - 10https://gerrit.wikimedia.org/r/142499 (https://bugzilla.wikimedia.org/67188) [19:22:48] ori: updated. I wonder however if puppet will order that correctly. [19:23:06] I don't know if requires implies an 'immediately after' relations, or just an 'after' relation [19:23:20] ori, did you at some point intentionall s/apache::mod::php/apache::mod::php5/g ? [19:23:26] Russia and Turkey.. are both on 2 continents.. but more of them is Asia [19:23:34] Or purge some files that might have contained apache::mod::php? [19:23:43] ^d: You didn't move those wikis to Cirrus, then? [19:23:43] old discussion whether they belong to Europe or not ..hmmm [19:23:45] yes, redid that whole hierarchy [19:23:52] andrewbogott: sorry, did i miss a reference? [19:23:56] shld be php5 [19:24:01] <^d> twkozlowski: No, we had to roll back. We'll be doing half that list on Monday instead. [19:24:04] yeah, but it's trivial to fix. Just wanted to make sure it was expected. [19:24:07] yep [19:24:42] ^d: Is there a patch I can link in Tech News? [19:25:01] <^d> That'd be https://gerrit.wikimedia.org/r/#/c/142590/ [19:25:39] Weee! Thanks :-) [19:25:52] <^d> yw [19:26:41] Ah, I was wondering how was load, gotta wait more [19:26:50] ottomata: yeah in the puppet statsd reporter [19:27:03] ottomata: what was your question about it? [19:27:31] ori, do you know which version of maxmind we're using? geolite? [19:27:45] (03CR) 10Ori.livneh: [C: 031] "nice! LGTM." [operations/puppet/nginx] - 10https://gerrit.wikimedia.org/r/142499 (https://bugzilla.wikimedia.org/67188) (owner: 10Yuvipanda) [19:27:48] welp, jsut not working for me :p [19:27:50] statsd [19:27:54] maybe I need to pulll.. [19:28:20] ottomata: that's in operations/puppet [19:28:28] ah [19:28:31] Eloquence: no, i think it's the city one.. one sec [19:28:33] i'm trying in vagrant right now [19:28:38] i see facts in ops puppet ja [19:29:05] ottomata: btw YuviPanda's change ^ is good if you feel mergy [19:29:20] :D [19:29:29] will submit a patch that does a submodule bump now [19:29:48] whoa crazy package names [19:29:52] those are in ubuntu/debian? [19:30:00] Eloquence: yes, GeoIPCity: https://github.com/wikimedia/operations-puppet/blob/production/templates/varnish/geoip.inc.vcl.erb#L30 [19:30:03] ottomata: yep [19:30:07] (03Draft2) 10Jackmcbarn: Let sysops edit the GWToolset namespace on Commons [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142617 (https://bugzilla.wikimedia.org/67209) [19:30:12] ori: let me verify that the -dbg versions are on [19:30:15] ottomata: ^ [19:30:18] k [19:30:28] ottomata: ori they are [19:30:40] they're not debug *versions*; just debug symbols [19:30:42] (03PS3) 10Jackmcbarn: Let sysops edit the GWToolset namespace on Commons [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142617 (https://bugzilla.wikimedia.org/67209) [19:30:57] ori: right. I meant the -dbg variant of the package names. [19:31:12] ugh, the -dbg variant of the package names built with the three package variants? [19:31:22] so much variants, reminds me of COM [19:31:39] ok cool [19:31:41] so good to go then [19:31:47] (03PS4) 10Scottlee: Fixed spacing and lint rules for manifests files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142479 [19:31:52] (03CR) 10Ottomata: [C: 032 V: 032] nginx: Make nginx package name configurable [operations/puppet/nginx] - 10https://gerrit.wikimedia.org/r/142499 (https://bugzilla.wikimedia.org/67188) (owner: 10Yuvipanda) [19:32:05] look at this little submodule getting some use :) [19:32:11] and improvement :) [19:32:19] whee :D [19:32:23] ottomata is there a packaging task that needs to be done? [19:32:24] ottomata: gonna port labsproxy to this soon [19:32:33] hm, ori, in vagrant, does pluginsync need to be set to true in puppet.conf for custom module facts to work? [19:32:39] YuviPanda: remember you need to update the submodule in operations/puppet [19:32:46] dogeydogey: hm, not that I know of off hand [19:32:51] ori: yeah, doing that momentarily [19:33:03] unless you happen to be very experienced with maven-deb-helper and building java deb packages in general [19:33:04] ottomata: merge https://gerrit.wikimedia.org/r/#/c/142500/ as well? just adds .gitreview for the nginx package [19:33:07] err, the nginx repo [19:33:22] ottomata: good question. i don't think so, since i test my puppet changes in vagrant, and i don't recall having to do that for modules with custom facts. but i'm not 100% sure. [19:33:23] (03PS2) 10Ottomata: Add .gitreview [operations/puppet/nginx] - 10https://gerrit.wikimedia.org/r/142500 (owner: 10Yuvipanda) [19:33:34] YES my stuff finally passed [19:33:36] there aren't any modules with custom facts in vagrant right now, are there? [19:33:39] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:41] (03CR) 10Ottomata: [C: 032 V: 032] Add .gitreview [operations/puppet/nginx] - 10https://gerrit.wikimedia.org/r/142500 (owner: 10Yuvipanda) [19:33:48] ottomata: no, but i would have tested puppet-statsd there, for example [19:33:52] hm [19:33:55] !log puppet-compiler: removed modules/varnish at root@puppet-compiler02:/opt/wmf/software/compare-puppet-catalogs/external/puppet and resetted repo. [19:33:59] Logged the message, Master [19:33:59] just to be sure i'm doing this right [19:34:12] i just have to put my fact .rb files in modules//ib/facter/ [19:34:13] right? [19:34:15] and then it should work? [19:34:21] (I know they work if I set FACTERLIB to that path) [19:34:22] (03PS1) 10Yuvipanda: nginx: Update nginx submodule [operations/puppet] - 10https://gerrit.wikimedia.org/r/142625 [19:34:26] (on the CLI) [19:34:32] doesn't work in puppet yet [19:34:39] yep [19:34:42] hm [19:34:44] k [19:34:54] ottomata: if it doesn't work, assume i was wrong [19:35:01] i can test, too, if you'd like a sanity check [19:35:12] ja would, see if you can make something work [19:35:17] doing so [19:35:20] danke [19:35:29] ottomata: ^ submodule bump. [19:36:18] ori: Any quick guess about what's happening here? http://puppet-testing-proxy.wmflabs.org (that's before your latest patch, but after a bunch of other ones :) ) [19:36:41] (03CR) 10Brian Wolff: "(Nitpicky) Most rights use a '-' not an '_' to separate words. More importantly, I think it would be better to split the rights up in the " [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142617 (https://bugzilla.wikimedia.org/67209) (owner: 10Jackmcbarn) [19:37:35] (03PS2) 10Ottomata: nginx: Update nginx submodule [operations/puppet] - 10https://gerrit.wikimedia.org/r/142625 (owner: 10Yuvipanda) [19:37:42] (03CR) 10Ottomata: [C: 032 V: 032] nginx: Update nginx submodule [operations/puppet] - 10https://gerrit.wikimedia.org/r/142625 (owner: 10Yuvipanda) [19:38:00] ottomata: ty [19:38:25] (03CR) 10Jackmcbarn: "The rights required to use the special page are already configurable in the extension." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142617 (https://bugzilla.wikimedia.org/67209) (owner: 10Jackmcbarn) [19:38:44] hmm, ori, YuviPanda. I'm getting + class nginx::ssl with this merge too [19:38:45] is that ok? [19:38:53] has that been reviewed properly? [19:39:01] that's _joe_'s change, dunno [19:39:24] andrewbogott: no quick guesses no, but i'd check the apache confs [19:39:32] hm, he's not online, its probably ok, its just adding the class [19:39:33] hm [19:39:49] ok, merged [19:40:06] hm, eyah, ori, i just tried with a non-module custom fact [19:40:12] i.e. puppet/lib/facter [19:40:16] didn't work either :/ [19:40:35] ottomata: it worked for me [19:40:45] hmm [19:40:46] ==> default: info: Loading facts in /tmp/vagrant-puppet-3/modules-0/factertest/lib/facter/hello_world.rb [19:40:46] ==> default: notice: Scope(Class[Factertest]): Hello, world! [19:41:07] oh… this is happening. apache2: Syntax error on line 222 of /etc/apache2/apache2.conf: No matches for the wildcard '*' in '/etc/apache2/sites-local', failing (use IncludeOptional if required) [19:41:07] ottomata: i'll submit it as a patch so you can see [19:41:11] So apache can't start [19:41:25] andrewbogott: d'oh, i'll fix, sec [19:41:27] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.016 second response time [19:41:37] dumb apache not allowing * to be an empty match [19:42:23] ottomata: https://gerrit.wikimedia.org/r/142627 [19:42:37] PROBLEM - Host radon is DOWN: PING CRITICAL - Packet loss = 100% [19:42:44] ottomata: sorry, lost connection. I was saying they weren't used anywhere [19:42:54] puppet 3! is your vagrant all upgraded or sumpin!? [19:43:02] (03PS1) 10Andrew Bogott: Replace apache::mod::php with apache::mod::php5 [operations/puppet] - 10https://gerrit.wikimedia.org/r/142629 [19:43:31] ottomata: no, vagrant just appends -#num if the /tmp dir already exists [19:43:45] oh ha [19:43:45] ok [19:43:52] vagrant@mediawiki-vagrant:~$ puppet --version [19:43:52] 2.7.11 [19:44:03] ACKNOWLEDGEMENT - Host radon is DOWN: PING CRITICAL - Packet loss = 100% daniel_zahn being switched to private IP [19:44:19] (03CR) 10Ori.livneh: [C: 031] Replace apache::mod::php with apache::mod::php5 [operations/puppet] - 10https://gerrit.wikimedia.org/r/142629 (owner: 10Andrew Bogott) [19:44:57] ok i'm going to pull that exact patch down and see what happens [19:46:19] yours worked! [19:46:43] (03PS1) 10Yuvipanda: dynamicproxy: Use nginx module [operations/puppet] - 10https://gerrit.wikimedia.org/r/142631 [19:46:55] testing it now [19:46:58] (03CR) 10Dzahn: [C: 032] switch radon from public to private IP [operations/dns] - 10https://gerrit.wikimedia.org/r/142611 (owner: 10Dzahn) [19:47:35] (03CR) 10Andrew Bogott: [C: 032] Replace apache::mod::php with apache::mod::php5 [operations/puppet] - 10https://gerrit.wikimedia.org/r/142629 (owner: 10Andrew Bogott) [19:48:05] (03PS2) 10Yuvipanda: dynamicproxy: Use nginx module [operations/puppet] - 10https://gerrit.wikimedia.org/r/142631 [19:48:20] ottomata: going to find a puppetmaster to test ^ now [19:48:58] (03PS1) 10Ori.livneh: role/labs: touch an empty conf file in sites-local [operations/puppet] - 10https://gerrit.wikimedia.org/r/142632 [19:49:06] ^ andrewbogott [19:49:26] ottomata: maybe a silly typo in the file path or the var names? [19:49:27] hmm bahhhh [19:49:31] ottomata: every has to match [19:49:33] everything [19:49:37] hm [19:49:50] (03CR) 10Andrew Bogott: [C: 032] "Silly apache :(" [operations/puppet] - 10https://gerrit.wikimedia.org/r/142632 (owner: 10Ori.livneh) [19:51:02] so... why doesn't DNS work on my instance? [19:51:09] https://wikitech.wikimedia.org/wiki/Nova_Resource:I-0000029f.eqiad.wmflabs [19:51:14] but on bastion: [19:51:22] lib/facter/XXX.rb, Facter.add('XXX'), notice($XXX) <-- 'XXX' has to match across these three [19:51:33] ottomata: ^ [19:52:16] cscott@bast1001:~$ host towtruck [19:52:16] Host towtruck not found: 3(NXDOMAIN) [19:52:16] cscott@bast1001:~$ nslookup towtruck.eqiad.wmflabs [19:52:16] Non-authoritative answer: [19:52:17] *** Can't find towtruck.eqiad.wmflabs: No answer [19:52:42] bast1001 is .eqiad.wmnet [19:52:48] different vlan [19:53:57] (03CR) 10Greg Grossmeier: [C: 04-1] "Just a ceremonial -1 until the bug/consensus discussion is resolved." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142617 (https://bugzilla.wikimedia.org/67209) (owner: 10Jackmcbarn) [19:55:41] cscott: if you want your instance to be reachable from the outside world, you need to use yuvipanda's dynamicproxy thing or assign the instance a public ip and wmflabs.org hostname [19:56:11] use the dynamicproxy thing, should work 99% of cases [19:56:53] just tried https://wikitech.wikimedia.org/wiki/Special:NovaProxy looks like it's working. pretty nice. [20:01:12] cscott: :) you get SPDY/2 and websocket proxying as well. [20:02:11] ! ori, i think my vm is doing weird stuff with the tmp or shared folder [20:02:30] my modules lib direcotry doesn't exist in the /tmp/vagrant-puppet dir when I run puppet! [20:02:37] time for vagrant reload... [20:02:56] andrewbogott: I'm repurposing dynamicproxy-test2 as a self hosted puppetmaster to test the nginx changes. any objections? [20:03:14] yuvipanda: nope, that's fine [20:03:21] andrewbogott: cool, ty [20:04:48] andrewbogott: is the labs puppetmaster ok? [20:04:51] err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class role::dynamicproxy for i-00000111.eqiad.wmflabs on node i-00000111.eqiad.wmflabs [20:04:52] warning: Not using cache on failed catalog [20:04:52] err: Could not retrieve catalog; skipping run [20:04:58] ran into similar issues on tools-proxy-test as well [20:05:47] yuvipanda: puppetmaster should be ok, maybe that's a real error [20:06:12] andrewbogott: hmm, error 400 [20:06:18] now it works ori! [20:06:19] ahhhh [20:07:10] ok, what bastion should i be using to get to towtruck.eqiad.wmflabs ? [20:07:20] and! [20:07:30] also, ori, facts do not need to match their filenames [20:07:37] you can put multiple facts in one file [20:07:51] cscott: bastion.wmflabs.org should work [20:08:27] yuvipanda: looks like that class is now called role::dynamicproxy:: [20:08:30] if you're adding a new public IP for a labs instance, that other labs instances might want to reach via its public hostname, you may want to update manifests/openstack.pp [20:08:37] andrewbogott: aah, hmm. [20:08:40] $nova_dnsmasq_aliases [20:12:20] ori: do you mind logging into puppet-testing and seeing what's wrong with my apache conf? It looks like everything is marked 'enabled' to me but… no dice. [20:13:16] (03PS1) 10Ottomata: Add missing templates [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142637 [20:13:50] (03CR) 10Ottomata: [C: 032 V: 032] Add missing templates [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142637 (owner: 10Ottomata) [20:14:23] (03PS1) 10Ottomata: Add custom facts for CDH versions [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142638 [20:14:29] ori, would you mind reviewing that last one for me real quick? ^ [20:14:48] it works, but I woudln't mind a second pair of eyes on it to say "hm, you can do this better in another way..." [20:15:03] Poor ori sure is going to have a lot of pings when he gets back from lunch [20:15:08] it basically just runs commands and parses the output to get version numbers [20:15:21] thougth it would be better to run commands rather than parse the output of apt or dpkg or whatever [20:15:23] not sure though [20:16:03] (03CR) 10Ori.livneh: [C: 031] "Haven't tested, but this looks right." [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142638 (owner: 10Ottomata) [20:16:19] andrewbogott: i have to run for 30 mins [20:16:21] i can look then [20:16:25] thanks [20:16:33] I will prod listelessly until then [20:16:39] listlessly? sp? [20:16:40] hm [20:17:14] (03CR) 10Ottomata: [C: 032 V: 032] Add custom facts for CDH versions [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142638 (owner: 10Ottomata) [20:18:27] (03PS1) 10Dzahn: switch radon from public to private IP in DHCP [operations/puppet] - 10https://gerrit.wikimedia.org/r/142646 [20:19:02] (03CR) 10Dzahn: [C: 032] switch radon from public to private IP in DHCP [operations/puppet] - 10https://gerrit.wikimedia.org/r/142646 (owner: 10Dzahn) [20:20:17] andrewbogott: btw [20:20:19] that instance is trusty [20:20:29] have you adapted that module for trusty? [20:20:32] if not, it may not work [20:20:45] precise is on apache 2.2, some things are different [20:20:45] hm, good point. I'll try a precise instance and see if it's the same [20:34:58] cmjohnson1 / papaul : I'm having to figure out the LVS fiber runs, which are the lc/lc singlemode fiber we have to connect to each lvs server and every row stack [20:35:16] what are you figuring out? [20:35:20] how long of a fiber do you think we need to route from the raceway above the racks, down to the lvs servers in each? [20:35:23] length? [20:35:23] 1 or 2M? [20:35:37] 1M could be too short i fear, 2M is just excess but that seems ok. [20:35:46] depends on where the lvs servers are on the rack [20:35:57] well, we are planning to put them in the top 1/4th i thought [20:35:58] if they're at the bottom...3M [20:36:00] for ease of this [20:36:10] okay top 1/4 then 2M [20:36:11] they're on order and havent arrived yet [20:36:28] i figure with it having each lvs server using 3 fibers plus a DAC [20:36:33] it seemed better to put in top of rack [20:36:41] sound feasible? [20:37:24] in which rack will it be? [20:37:36] papaul: all the lvs servers will be in a2/a7/b2/b7 [20:37:45] ok [20:37:47] 2 in a2, 2 in a7, 1 in b2, 1 in b7 [20:38:03] so im estimating the raceway from a2 to b2 is 18 feet [20:38:12] counting the tiles (24") [20:38:17] on the layout diagram [20:38:26] (but you both should look and also see if that makes sense to you) [20:38:44] ok i am checking [20:39:59] seems like its a2 to a4 (2), then a4 to b4 (5), then b4 to b2 (2) for 9 tiles [20:40:07] Rob and Chris 18 feet will be find [20:41:37] a2 to b4 close to 12 feet [20:41:47] sorry a2 to a4 [20:42:41] a4 to b4 close to 18 feet [20:43:56] heya ori, you back? [20:44:15] whatcha think about putting include role::mediawiki inside of vagrant-managed.pp by default? [20:44:21] that way it can be disable via CLI [20:44:29] for hadoop stuff it doesn't need included [20:44:38] and just installs a lot of stuff hadoop doesn't need [20:45:00] yuvipanda: i think we had a 'greenlake' host, too, which ran some nodejs stuff? i don't think it successfully made the move to eqiad. [20:45:12] cscott: ah, yeah, we had greenlake. [20:45:20] Robh: if we have excess fiber it can be spooled tother and sit in the raceways [20:45:27] cscott: and pinklake too, IIRC [20:45:28] yep [20:45:32] and there will be [20:45:44] cmjohnson1: I'm going to assign this ticket to you to review when i finish it in a few minutes [20:45:55] just to reality check the lengths and quantity [20:45:59] yes, the pinklake.wmflabs url redirection seems to have gone away, but i recreated it with Special:NovaProxy (as togetherjs.wmflabs.org) [20:46:18] yuvipanda: but i don't remember where greenlake used to point to. i can't find a corresponding instance any more. [20:46:18] cscott: cool! :) [20:46:25] cscott: yeah, I don't either. [20:46:40] did stuff which didn't make the move to eqiad get archived anywhere? [20:46:53] cscott: andrewbogott might know. [20:46:55] (03PS1) 10Ottomata: Use mapreduce.shuffle instead of mapreduce_shuffle as comes with default yarn-site.xml [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/142706 [20:46:57] if not, it's probably straightforward for me just to recreate the host. [20:46:57] (03PS1) 10Ottomata: Updates for TODO.md [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/142707 [20:46:59] (03PS1) 10Ottomata: Use exec to make sure hosts.exclude exists rather than managing file resource [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/142708 [20:47:01] (03PS1) 10Ottomata: Add TODO for fixing Oozie Server extjs install [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/142709 [20:47:03] (03PS1) 10Ottomata: Use mapreduce_shuffle! This is correct [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/142710 [20:47:05] (03PS1) 10Ottomata: Add missing templates [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/142711 [20:47:06] Whoa [20:47:07] (03PS1) 10Ottomata: Add custom facts for CDH versions [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/142712 [20:47:08] AHHHHH [20:47:09] (03PS1) 10Ottomata: Use dpkg-query to get cdh version facts [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/142713 [20:47:11] stupid wrong remote [20:47:27] yuvipanda: unless you or marktraceur convinced me to use puppet to set up greenlake, in which case the config is probably in some repo somewhere still. [20:47:27] andrewbogott: now I'm just getting 'err: Could not retrieve catalog from remote server: Connection refused - connect(2)' [20:47:40] andrewbogott: same as I was getting in tools-proxy-test [20:47:43] cscott: pretty much all instances were moved, just some of them are in a shutdown state. [20:47:43] cscott: don't think we did, no. [20:47:56] yuvipanda: is that with self-hosted puppet? If so you just need to restart your puppet master. [20:47:59] (03PS1) 10Ottomata: Use dpkg-query to get cdh version facts [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142715 [20:48:17] (03Abandoned) 10Ottomata: Use mapreduce.shuffle instead of mapreduce_shuffle as comes with default yarn-site.xml [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/142706 (owner: 10Ottomata) [20:48:20] andrewbogott: ah, doing that now [20:48:23] (03Abandoned) 10Ottomata: Updates for TODO.md [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/142707 (owner: 10Ottomata) [20:48:26] (03Abandoned) 10Ottomata: Use exec to make sure hosts.exclude exists rather than managing file resource [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/142708 (owner: 10Ottomata) [20:48:29] (03Abandoned) 10Ottomata: Add TODO for fixing Oozie Server extjs install [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/142709 (owner: 10Ottomata) [20:48:33] (03Abandoned) 10Ottomata: Use mapreduce_shuffle! This is correct [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/142710 (owner: 10Ottomata) [20:48:35] andrewbogott: hmm, did a service puppet restart, same problems [20:48:36] (03Abandoned) 10Ottomata: Add missing templates [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/142711 (owner: 10Ottomata) [20:48:41] (03Abandoned) 10Ottomata: Add custom facts for CDH versions [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/142712 (owner: 10Ottomata) [20:48:45] (03Abandoned) 10Ottomata: Use dpkg-query to get cdh version facts [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/142713 (owner: 10Ottomata) [20:48:47] yuvipanda: service puppetmaster restart [20:49:01] andrewbogott: hmm, puppetmaster unrecognized service [20:49:06] andrewbogott: I think there were errors in the initial run as well [20:49:10] oh, is this precise or trusty? [20:49:48] andrewbogott: precise [20:49:53] oh, hm. [20:50:00] andrewbogott: dynamicproxy-test2 [20:50:21] (03CR) 10Gage: [C: 032] "Looks good! Discussed on IRC." [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142715 (owner: 10Ottomata) [20:51:38] (03PS1) 10Ottomata: Fix .gitreview url [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142717 [20:51:52] (03CR) 10Ottomata: [C: 032 V: 032] Fix .gitreview url [operations/puppet/cdh] - 10https://gerrit.wikimedia.org/r/142717 (owner: 10Ottomata) [20:52:35] yuvipanda: I don't immediately know what's happening there. It looks like the puppetmaster isn't running or answering but I know not why. Usually that shows up as a cert problem [20:52:52] andrewbogott: right. any idea how to fix, considering it's happened on two instances in the last few hours? [20:53:04] which was the other one? [20:53:17] andrewbogott: tools-proxy-test [20:53:36] exact same issues. some errors that scrolled off too quickly when running initial puppet agent, anda then this [20:54:55] ok [20:57:27] andrewbogott: re trusty things [20:57:34] is trusty ready to use on prod nodes? [20:57:40] i'm gongi to be reinstalling the hadoop cluster soon [20:58:19] ottomata: yeah, it pretty much is. It's running a few things. [20:58:29] But, you'll be an early adopted, so keep an eye out and report back with your findings :) [21:00:22] hmm, how do I even tell it to install? [21:00:43] oh pff [21:00:48] cloudera doesn';t have trusty pacakges [21:00:50] http://archive-primary.cloudera.com/cdh5/ubuntu/ [21:00:55] so maybe not! [21:00:55] papaul: so looking at the mc servers in a2 [21:01:14] do you think they need a 1m or 2m fiber to route along the fiber organizer we're ordering to go vertically [21:01:40] i think 2m is needed for routing but 1m may possibly work. [21:01:55] PROBLEM - MySQL Processlist on db1068 is CRITICAL: CRIT 84 unauthenticated, 0 locked, 0 copy to table, 0 statistics [21:02:55] RECOVERY - MySQL Processlist on db1068 is OK: OK 1 unauthenticated, 0 locked, 0 copy to table, 1 statistics [21:03:03] yuvipanda: i found greenlake in my irc logs. turns out it was a redirection to port 8001 of the same instance. [21:03:14] can i do that with novaproxy? [21:03:35] cscott: not presently, no :( the proxy supports it but it's not exposed in the interface. [21:03:41] andrewbogott: ^ can we add that as a field to wikitech? [21:04:08] yuvipanda: sorry, can you start at the beginning? [21:04:43] andrewbogott: ah. so right now the wikitech NovaProxy interface sets all proxies to go to port 80. cscott wants that to be configurable from the interface [21:04:59] Oh, I see. I thought it maybe was, if you just stick a : in the field? [21:05:04] If not, we can add it. Just make me a bug? [21:06:34] wait, it might be there already. https://wikitech.wikimedia.org/w/index.php?title=Special:NovaProxy&action=create&project=visualeditor®ion=eqiad lets me specify an instance port. [21:08:10] hah I knew it! [21:09:00] andrewbogott: cscott oh, cool. [21:11:54] yes [21:11:54] RobH :yes [21:12:13] andrewbogott: back, still need me to look at something? [21:12:30] oooh, morebots [21:12:35] ori: maybe? I suspect that Trusty is at fault, but I'm still interested in fixing it. [21:12:52] ori: quick question for HHVM: have you tried running PHPUnit tests with the extensions deployed on the cluster? [21:13:05] hashar: yes [21:13:06] ori: the instance is puppet-testing.eqiad.wmflabs and you can hit it via http://puppet-testing-proxy.wmflabs.org [21:13:21] ori: I tried a bit yesterday with plain PHP and there is a lot of random failures and incompatibilities. I filled a few bugs already [21:13:34] yeah, me too. no incompatibilities, though. [21:14:01] I have stuff like https://integration.wikimedia.org/ci/job/mediawiki-core-extensions-integration/42/console [21:14:11] yuvipanda, andrewbogott: ... but i'm getting a 502 "bad gateway" error when I try to hit hub.wmflabs.org (which should redirect to 8080 on towtruck.eqiad) [21:14:25] cscott: is 8080 open in your 'security groups'? [21:14:28] cscott: is the firewall open on 8080? [21:14:30] cscott: for your towtruck project? [21:14:45] good question. i assume i would have opened it before, but i'll double check. [21:14:54] lot of errors / failures and eventually ends up with a fatal hehe [21:14:55] hashar: https://dpaste.de/U6Cx/raw [21:15:05] maybe I should write a vagrant role to easily reproduce a setup with multiple extensions and challenge folks to make the tests to pass [21:15:12] the if ( $loadKnownBad ) { } contains the extensions I know are failing [21:15:41] ori: is that from mediawiki/vagrant ? [21:15:48] hashar: no, just a copy of a local file i have on osmium [21:15:55] ah [21:15:58] hashar: you can ssh there btw, it should be accessible to you [21:16:11] we should join our efforts so :] [21:16:24] the success of certain unit tests is mutually exclusive with the success of others [21:16:28] which is highly problematic [21:16:29] I wanted to check the phpunit tests to later run them on HHVM to help push hhvm [21:16:51] (03PS5) 10Withoutaname: Initialize some settings for wikimania 2015 wiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/139279 (https://bugzilla.wikimedia.org/66370) [21:16:51] for example, ProofreadPage adds prp-pagequality=".." attributes to elements [21:16:58] which obviously makes a whole bunch of parser tests in core fail [21:16:59] yuvipanda: I just started a new self-hosted instance and it works fine. So, that teaches us nothing... [21:17:23] ori: what I do is clone all extensions then load their default entry point in whatever order some PHP script yield them: https://github.com/wikimedia/integration-jenkins/blob/master/mediawiki/conf.d/50_mw_ext_loader.php [21:17:33] andrewbogott: ah, hmm. true. [21:17:37] andrewbogott: where is this? maybe I can use this :) [21:17:51] yuvipanda: not until I'm done with it [21:18:01] andrewbogott: ah :D ok, how long do you expect that to take? [21:18:03] yuvipanda: but, suggests that if you make a fresh instance it'll just work [21:18:06] hashar: *nod*, but then you run just the extension tests, right? [21:18:08] So, maybe that's a quick fix? [21:18:17] andrewbogott: hmm, true. let me make one then. [21:18:19] Still interested in why those other boxes degraded though [21:18:31] ori: for now yes: cd tests/phpunit; php phpunit.php --testsuite extensions [21:18:47] ori: have you filled bug for each issues appearing on your paste https://dpaste.de/U6Cx/raw ? [21:18:51] hashar: not every extension declares its tests with @group extensions [21:18:54] hashar: almost all of them [21:19:04] the bugzilla urls should be in the comments [21:19:04] andrewbogott: yeah, ok [21:19:23] ori: ah yeah indeed. I am missing syntax highlighting [21:19:37] hashar: i fixed a whole bunch of small ones too, just look at extension commits in https://gerrit.wikimedia.org/r/#/q/owner:self,n,z [21:19:39] cscott: did that work? [21:19:41] er, not self [21:19:50] yuvipanda: still clicking through the config [21:19:51] https://gerrit.wikimedia.org/r/#/q/owner:%22Ori.livneh+%253Cori%2540wikimedia.org%253E%22,n,z rather [21:20:00] cscott: heh, ok :) [21:20:16] how do i put the towtruck instance in my new 'togetherjs' security group? [21:20:21] ori: the --testsuite extensions does not rely on @groups annotations. It uses the UnitTestFile (or something) hook [21:20:42] cscott: sadly, you can't. I suggest modifying the base one [21:20:50] (03PS9) 10Withoutaname: Delete ve.wikimedia.org and leave redirect [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/131907 (https://bugzilla.wikimedia.org/55737) [21:20:51] hashar: ah, that's good. it must be that some extensions don't register the appropriate handler, then, because i noticed that it fails to execute the tests of certain extensions [21:21:08] yuvipanda: (!) [21:21:19] cscott: inorite [21:24:08] ori: that is possible [21:24:35] ori: seems we need a major push forward. We can probably involves our community to get the tests to pass / fix the issues. Much of them are probably easy to handle [21:24:44] it is just there is a lot of the [21:24:45] m [21:28:03] Hm, nopes, ori: same Apache privs error on a precise box. [21:30:38] andrewbogott: hmm, interesting. I created a new instance, and seemed to be able to login, but everytime I try to execute something, I get [21:30:38] sudo: unknown uid 2029: who are you? [21:30:46] and now I can't ssh to it anymore (unknown publickey) [21:30:49] * yuvipanda is confused [21:31:08] well, I've never seen that before! What instance is it? [21:31:19] And are you maybe logged in before puppet has figured out who you are? [21:31:37] andrewbogott: right, but usually puppet never lets me log in [21:31:46] andrewbogott: damnamicproxy-test [21:32:01] you must've caught it at /just/ the right time… [21:32:19] andrewbogott: heh [21:32:42] yuvipanda: I can't log in at all. [21:32:50] andrewbogott: yeah, ame here atm [21:32:51] *same [21:33:10] yuvipanda: maybe just scrap it and try again :( [21:33:20] andrewbogott: :( ok [21:35:24] andrewbogott: created feelgoodproxy-test.eqiad.wmflabs, let me wait a bit [21:37:17] andrewbogott: http://puppet-testing-proxy.wmflabs.org/wiki/Main_Page [21:37:36] andrewbogott: but still debugging something, hang on [21:39:51] ori-l_: I take it you changed something, it wasn't like that when you got there? [21:40:20] andrewbogott: apache 2.4 changed the way access directives work: http://httpd.apache.org/docs/2.4/upgrading.html#access [21:40:34] andrewbogott: changing 'allow from all' to 'require all granted' works [21:40:37] however [21:41:13] apache 2.4 ships with a module, mod_access_compat, which allows you to use the old config directives [21:41:17] and we enabled it by default [21:41:20] and it is enabled on that host [21:41:24] so i'm a bit perplexed [21:41:28] http://httpd.apache.org/docs/2.4/mod/mod_access_compat.html [21:42:02] andrewbogott: this one seems ok :) I think instances don't like having the word 'damn' in their names [21:42:14] yuvipanda: that's probably it [21:43:02] grumble, can't seem to get parsoid running on towtruck [21:49:40] (03PS3) 10Yuvipanda: dynamicproxy: Use nginx module [operations/puppet] - 10https://gerrit.wikimedia.org/r/142631 [21:54:18] ports and proxies hate me [21:56:03] cscott: ? [21:56:45] i can't seem to get parsoid running on port 5666 and the hub running on port 8080, which is my desired config. [21:57:05] the instance security group should have port 5666 open at least, but that doesn't seem to be working [21:57:10] not even on localhost with curl [21:57:19] cscott, what host and project? [21:59:18] andrewbogott: towtruck.eqiad.wmflabs, https://wikitech.wikimedia.org/wiki/Nova_Resource:I-0000029f.eqiad.wmflabs [21:59:25] andrewbogott: so, tested https://gerrit.wikimedia.org/r/#/c/142631/, seems to work. wanna merge? :) [22:00:14] aww [22:00:17] ottomata1: can you? ^ [22:00:27] ottomata1: it's the nginx module change for the proxies [22:00:45] (03PS1) 10Ori.livneh: mediawiki_singlenode: apache 2.4 compat [operations/puppet] - 10https://gerrit.wikimedia.org/r/142730 [22:00:52] andrewbogott_afk: ^ gratis [22:01:05] (03PS1) 10Yuvipanda: dynamicproxy: Enable diamond collector for nginx [operations/puppet] - 10https://gerrit.wikimedia.org/r/142732 [22:01:18] chasemp: ^ once https://gerrit.wikimedia.org/r/#/c/142631/ gets merged I'll get nginx stats on! \o/ [22:01:54] nice [22:02:08] chasemp: think you can CR? I tested https://gerrit.wikimedia.org/r/#/c/142631/ with a self hosted puppetmaster as well [22:03:33] cscott: sorry, got kicked off wifi [22:03:59] andrewbogott: not sure if you got my message, but https://gerrit.wikimedia.org/r/#/c/142631/ has been tested if you want to merge :) [22:04:26] yuvipanda: you linked it no fewer than four times in three minutes [22:04:34] I should probably sleep, ori-l_ [22:04:35] thank you ori! [22:04:44] ori-l_: but ^ doesn't really work :| [22:04:45] (03CR) 10Andrew Bogott: [C: 032] mediawiki_singlenode: apache 2.4 compat [operations/puppet] - 10https://gerrit.wikimedia.org/r/142730 (owner: 10Ori.livneh) [22:04:46] thereby contesting my "puppet merge nag" champion title [22:04:53] ori-l_: hah! :) [22:04:56] ori-l_: at least you can self merge :) [22:05:03] kinda sorta [22:05:08] ori-l_: still. [22:05:11] ori-l_: do you keep records? [22:05:38] when folks just stop responding i usually assume i've crossed some boundary of good taste [22:05:46] ori-l_: yeah, I think I did that too here [22:05:53] * yuvipanda apologies to those involved without pinging [22:06:03] being excited is still permitted tho :P [22:06:23] ori-l_: :D labs had no graphs, so when things went down we had no idea if it was because of load. this should help us fix that. [22:06:42] ori-l_: I should implement a limechat plugin that does 'ping quotas' [22:06:51] * andrewbogott has by now totally forgotten what he started out trying to do [22:07:39] ori_: I think I'm extra pingy today since I realized I just crossed 100 merged patches in the ops repo. [22:08:00] yuvipanda: I'm not ignoring you :) wrapped up in a convo elsewhere [22:08:06] puppet centurion [22:08:17] oh, i think i figured out why my ports weren't working. was reading the security group firewall rules backwards, so i was trying to use the one port that was totally blocked. [22:08:21] chasemp: yeah, I figured :) But I've been a bit pingy today :) [22:08:52] ori_: :) [22:09:22] manybubbles: https://gerrit.wikimedia.org/r/#/c/140567/ is already live? [22:11:13] (03PS4) 10Ori.livneh: mediawiki_singlenode: port apache::vhost to apache::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/142206 [22:12:18] I'm trying to create a new project in gerrit and I'm getting "Capability createProject is required to access this resource" [22:12:32] I can haz createProject capability? [22:12:35] Oh yeah, that [22:13:04] twentyafterfour: I've been told to ask ^d previously [22:13:18] somewhere in the wiki is request procedure :D [22:13:23] trying to create it within operations/debs [22:14:21] chasemp, twentyafterfour: request: can you port the apache::vhost resource in the new phabricator module to use apache::site instead? i'm trying to get rid of apache::vhost and i have patches for all except that one and one of the statistics sites [22:14:41] it's not at all time-sensitive, just whenever you have the chance [22:14:51] yes, I will do that ori [22:14:55] but probably monday? [22:15:01] * yuvipanda goes off to sleep. [22:15:03] and I will bug you about it when I'm confused :) [22:15:11] awesome thanks [22:15:16] bug away! [22:15:24] or ^demon|away [22:17:51] I can't find anything on wiki about requesting permission [22:18:17] there's https://www.mediawiki.org/wiki/Git/New_repositories/Requests [22:20:48] that's the one I know I think [22:20:53] * twentyafterfour is not impressed [22:20:57] but I thought it was on wikitech...:) [22:22:00] YuviPanda|zz, andrewbogott: ok, parsoid+ve is working, but special novaproxy is choking on websocket connections [22:24:19] requested [22:26:41] <^demon|away> twentyafterfour: Done. [22:26:54] thanks ^demon|away [22:27:37] <^demon|away> yw [22:30:44] So greg-g, authenticating with OAuth using mediawiki.org is currently broken (due to my namespace patch) [22:31:16] The fix is already merged, but wondering if it's worth deploying that this afternoon [22:31:40] csteipp: uhh, yeah, probably, soon, so it's not too late [22:31:47] csteipp: I assume this means all auth, not just new auth [22:31:59] like, just just for new apps, but for all app authentication [22:32:18] Yeah, all OAuth consumers that are doing authentication correctly. We just ran into it registering new phabricator accounts. [22:32:39] Alright, Let me do that... [22:33:49] * greg-g nods [22:37:14] creating friday work for greg-g, weekly mission accomplished [22:37:36] chasemp: don't advertise it [22:37:45] PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Fri 27 Jun 2014 20:37:28 UTC [22:38:10] chasemp: go take care of that db puppet freshness! :P [22:39:27] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Fri Jun 27 22:39:19 UTC 2014 [22:39:49] boom [22:40:13] :( [22:40:29] you win this round, mr chase [22:43:15] wow, submodule updates when I haven't deployed (publicly) in a while takes forever... [22:46:18] yes.. [22:57:48] !log csteipp Synchronized php-1.24wmf11/extensions/OAuth/frontend/specialpages/SpecialMWOAuth.php: Fix OAuth Logins for wmf11 (duration: 00m 18s) [22:57:53] Logged the message, Master [23:00:05] greg-g: I'm done. Thanks! [23:00:55] good deal [23:24:41] (03PS8) 10MZMcBride: Improve nginx TLS/SSL settings. [operations/puppet] - 10https://gerrit.wikimedia.org/r/132393 (https://bugzilla.wikimedia.org/53259) (owner: 10JanZerebecki) [23:25:38] thanks csteipp and greg-g [23:26:00] (03PS1) 1020after4: initial commit [operations/debs/php-mailparse] (review) - 10https://gerrit.wikimedia.org/r/142751 [23:30:10] (03CR) 10Ori.livneh: "> In tests, this has shown some increase in server load and decrease in throughput, but nothing worrisome." [operations/puppet] - 10https://gerrit.wikimedia.org/r/132393 (https://bugzilla.wikimedia.org/53259) (owner: 10JanZerebecki) [23:44:20] (03PS9) 10Nemo bis: Improve nginx TLS/SSL settings. [operations/puppet] - 10https://gerrit.wikimedia.org/r/132393 (https://bugzilla.wikimedia.org/53259) (owner: 10JanZerebecki)