[00:00:00] MaxSem: and if so, can it switch to using /data/scratch instead? [00:00:41] 6Labs, 10Analytics, 13Patch-For-Review: Kill all unnecessary NFS usage in the analytics project - https://phabricator.wikimedia.org/T128804#2086918 (10yuvipanda) 5Open>3Resolved All usage has been deemed unnecessary! It is gone now! \o/ [00:01:00] go ahead, nuke it [00:01:13] MaxSem: the whole project? [00:01:17] MaxSem: or just NFS? [00:01:23] fs [00:01:26] nfs [00:01:32] ah [00:01:44] MaxSem: do you want me to just get rid of NFS or do you want me to do something specific [00:02:21] you shoud've warned you have THIS MANY questions! :} [00:02:27] 6Labs, 10Labs-Sprint-102, 10Labs-Sprint-103, 10Labs-Sprint-104, and 3 others: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#2086926 (10yuvipanda) [00:02:29] 6Labs, 6Discovery, 10Maps, 13Patch-For-Review: Investigate and reduce NFS use in maps-team project - https://phabricator.wikimedia.org/T103757#2086924 (10yuvipanda) 5Resolved>3Open Re-opening, since we can maybe get rid of 'em all! [00:02:30] MaxSem: :P https://phabricator.wikimedia.org/T103757 [00:03:53] 6Labs, 10Labs-Sprint-102, 10Labs-Sprint-103, 10Labs-Sprint-104, and 3 others: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#2086932 (10MaxSem) [00:03:55] 6Labs, 6Discovery, 10Maps, 13Patch-For-Review: Investigate and reduce NFS use in maps-team project - https://phabricator.wikimedia.org/T103757#2086930 (10MaxSem) 5Open>3declined Please don't take away scratch from us (or allow to create far, far larger partitions). [00:04:35] MaxSem: bah, nvm, I thought you were using /data/project not /scratch [00:04:37] nvm [00:04:56] 6Labs, 6Discovery, 10Maps, 13Patch-For-Review: Investigate and reduce NFS use in maps-team project - https://phabricator.wikimedia.org/T103757#2086940 (10yuvipanda) Bah, I had misread it and assumed they were using /data/proejct and not /data/scratch. sorry! [00:05:13] 6Labs, 10Labs-Sprint-102, 10Labs-Sprint-103, 10Labs-Sprint-104, and 3 others: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#2086943 (10yuvipanda) [00:05:15] 6Labs, 6Discovery, 10Maps, 13Patch-For-Review: Investigate and reduce NFS use in maps-team project - https://phabricator.wikimedia.org/T103757#2086942 (10yuvipanda) 5declined>3Resolved [00:07:48] mutante: ping? [00:07:51] are you around? [00:09:01] 6Labs, 10Labs-project-wikistats: Get rid of NFS from the wikistats project - https://phabricator.wikimedia.org/T128816#2086952 (10yuvipanda) [00:11:24] 6Labs, 10Labs-project-wikistats: Get rid of NFS from the wikistats project - https://phabricator.wikimedia.org/T128816#2086966 (10yuvipanda) Check with @dzahn [00:11:34] mutante: just wanted to ping about https://phabricator.wikimedia.org/T128816 :) [00:15:42] 6Labs, 10Labs-project-wikistats: Get rid of NFS from the wikistats project - https://phabricator.wikimedia.org/T128816#2087001 (10Dzahn) since recently, wikistats now clones from git and deploys with rsync. The NFS is not needed anymore. [00:15:47] yuvipanda: you can remove it [00:15:57] it's just that /project mount , right [00:16:23] i'm not deploying that with a .deb anymore since recently [00:16:49] the repo is still in the /debs/ namespace but i let puppet git pull it directly [00:16:54] mutante: yeah [00:16:56] mutante: awesome [00:16:58] mutante: thanks :D [00:17:07] no problem, yw [00:20:56] 6Labs, 10Labs-project-wikistats: Get rid of NFS from the wikistats project - https://phabricator.wikimedia.org/T128816#2087022 (10yuvipanda) w00t! [00:27:54] 6Labs, 10Labs-Sprint-102, 10Labs-Sprint-103, 10Labs-Sprint-104, and 3 others: Audit projects' use of NFS, and remove it where not necessary - https://phabricator.wikimedia.org/T102240#2087039 (10yuvipanda) [00:27:56] 6Labs, 10Labs-project-wikistats, 13Patch-For-Review: Get rid of NFS from the wikistats project - https://phabricator.wikimedia.org/T128816#2087036 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Is gone! Thanks :S [01:33:26] 10Tool-Labs-tools-stewardbots: hat-web-tools import for stewardbots - https://phabricator.wikimedia.org/T128743#2087214 (10Rschen7754) I believe everything was copied over to the stewardbots project before I stepped down, though unfortunately I don't remember the details and I suppose it could have gotten broken... [02:46:49] 6Labs, 10Tool-Labs, 10Mail, 6Operations: remove toolserver mail aliases - https://phabricator.wikimedia.org/T127543#2087339 (10Dzahn) p:5Normal>3Low [05:44:03] 6Labs, 10Wikimedia-Site-Requests, 10wikitech.wikimedia.org: powered by logos in footer have broken images on wikitech - https://phabricator.wikimedia.org/T128832#2087490 (10jeremyb-phone) [05:45:34] 6Labs, 10Wikimedia-Site-Requests, 10wikitech.wikimedia.org: powered by logos in footer have broken images on wikitech - https://phabricator.wikimedia.org/T128832#2087503 (10jeremyb-phone) [05:45:36] 6Labs, 10wikitech.wikimedia.org, 13Patch-For-Review: 404 poweredby_mediawiki_88x31.png when viewing wikitech - https://phabricator.wikimedia.org/T128747#2087504 (10jeremyb-phone) [06:08:10] 6Labs, 6Operations: Can't create account "Bishoy Camel" (user with a former SVN account not migrated) - https://phabricator.wikimedia.org/T128833#2087511 (10BishoyCamel) [06:18:58] 6Labs, 6Operations: Can't create account "Bishoy Camel" (user with a former SVN account not migrated) - https://phabricator.wikimedia.org/T128833#2087511 (10Peachey88) Hi @BishoyCamel, Can you please provide the following information so a labs admin can work on the issue: Preferred wikitech username: Prefer... [08:55:31] 6Labs, 10Tool-Labs: Remove or prune cdnjs on tools-static - https://phabricator.wikimedia.org/T128841#2087837 (10valhallasw) [08:57:24] 6Labs, 10Tool-Labs: Remove or prune cdnjs on tools-static - https://phabricator.wikimedia.org/T128841#2087850 (10Harej) If you remove Semantic UI, please let me know so that I can create a fork without webfonts. [09:21:40] 6Labs, 6Operations: Can't create account "Bishoy Camel" (user with a former SVN account not migrated) - https://phabricator.wikimedia.org/T128833#2087889 (10Aklapper) a:5RyanLane>3None [09:22:41] 6Labs, 6Operations: Can't create account "Bishoy Camel" (user with a former SVN account not migrated) - https://phabricator.wikimedia.org/T128833#2087511 (10Aklapper) (Removing task assignee as per [[ https://mediawiki.org/wiki/How_to_report_a_bug | guidelines ]].) [09:51:39] valhallasw`cloud: are you able to kick grrit-wm? [09:52:15] 6Labs, 10Tool-Labs, 7Software-Licensing: Remove or prune cdnjs on tools-static - https://phabricator.wikimedia.org/T128841#2087921 (10Legoktm) [09:53:29] valhallasw`cloud: nvm, someone else is going to [10:43:53] 6Labs, 10Tool-Labs, 7Software-Licensing: Remove or prune cdnjs on tools-static - https://phabricator.wikimedia.org/T128841#2087837 (10scfc) My understanding was that cdnjs.com provides only open source. Have those issues been reported upstream and declined? [12:05:22] 6Labs, 10Labs-project-wikistats: Get rid of NFS from the wikistats project - https://phabricator.wikimedia.org/T128816#2088222 (10Ricordisamoa) [12:16:51] 6Labs, 10Tool-Labs, 7Software-Licensing: Remove or prune cdnjs on tools-static - https://phabricator.wikimedia.org/T128841#2087837 (10Ricordisamoa) Bootswatch loads from fonts.googleapis.com too: https://tools-static.wmflabs.org/cdnjs/ajax/libs/bootswatch/3.3.6/flatly/bootstrap.min.css [13:45:28] 13:43 < icinga-wm> PROBLEM - Disk space on labvirt1008 is CRITICAL: DISK CRITICAL - free space: /var/lib/nova/instances 90264 MB (3% inode=99%) [13:45:36] ^ seems to be flapping all morning since I've been around [13:46:04] maybe-relatedly, there's still an icinga outstanding critical on labstore1002 for "DISK CRITICAL - free space: /dev 0 MB (3% inode=99%): ", but with alerts silenced [14:09:57] the labstore1002 thing isn't related, and should have cleared I'll look at it [14:11:21] yuvipanda: just a note to keep in mind scratch is relatively small comparatively [15:56:29] bd808: Are you here? [16:43:51] 10Wikibugs: Wikibugs doesn't shows duplicate status - https://phabricator.wikimedia.org/T128868#2088724 (10Luke081515) [16:46:10] 10Tool-Labs-tools-stewardbots: Make elections.php work again - https://phabricator.wikimedia.org/T128742#2084516 (10Glaisher) Can you check the error logs? Empty page usually means fatal error. [16:56:21] SMalyshev: Can wdq-beta tolerate some downtime? I’d like to migrate it to another host, which might take an hour or two. [16:58:54] http://tools.wmflabs.org/watroles/project/ores used to show the instances in the project ores [16:59:04] now it shows a 500 Internal Server Error [16:59:16] shall i make a ticket? [16:59:23] mutante: just for ores, or is watroles broken altogether? [16:59:46] hmm. just for ORES it seems [17:00:18] mutante: ok, a simple restart won’t fix it then… so yeah, ticket is probably in order. [17:00:26] http://tools.wmflabs.org/watroles/project/ores vs http://tools.wmflabs.org/watroles/project/wikistats [17:00:45] andrewbogott: ok, i will do that [17:02:40] 6Labs, 6Operations: labs precise instance not accessible after provisioning - https://phabricator.wikimedia.org/T117673#2088781 (10fgiunchedi) I've ran into this again just now with `monitoring-prometheus` instance which I've deleted a while ago but tried to reprovision just now. note the instance in this case... [17:02:53] 6Labs, 6Operations: labs precise and jessie instance not accessible after provisioning - https://phabricator.wikimedia.org/T117673#2088782 (10fgiunchedi) [17:06:07] now i wonder which phab tags are best for "watroles" itself [17:06:21] counts as labs infra? [17:06:38] something toollabs because thats where it runs? [17:07:12] some tools in toollabs have their own tags, but just a few [17:07:38] 6Labs, 10Labs-Infrastructure, 10Tool-Labs: watroles/ores returns 500 Internal Server Error, works for other projects - https://phabricator.wikimedia.org/T128871#2088804 (10Dzahn) [17:07:49] 6Labs, 10Labs-Infrastructure, 10Tool-Labs, 10ores: watroles/ores returns 500 Internal Server Error, works for other projects - https://phabricator.wikimedia.org/T128871#2088817 (10Dzahn) [17:08:51] 6Labs, 10Labs-Infrastructure, 10Tool-Labs, 10ores: watroles/ores returns 500 Internal Server Error, works for other projects - https://phabricator.wikimedia.org/T128871#2088804 (10Dzahn) I am reporting this because i wanted to review/test https://gerrit.wikimedia.org/r/#/c/270102/ and for that i wanted t... [17:14:41] 10Tool-Labs-tools-stewardbots: Make elections.php work again - https://phabricator.wikimedia.org/T128742#2088837 (10Glaisher) Nevermind, I just ran it locally. There's a syntax error there. The code is in a really messy state and I don't want to look into it now but I noticed these issues as well: ``` Notice: Un... [17:21:52] Is this the right place to complain about Phabricator being down? [17:22:38] Nah, -ops is. [17:27:23] andrewbogott: give me a min to check [17:28:02] andrewbogott: it's still in the middle of data load, can it wait for a bit? [17:28:12] SMalyshev: sure, ping me when ready? [17:28:18] andrewbogott: ok [17:28:24] thank you! [17:32:40] 6Labs, 10Labs-Infrastructure, 10Tool-Labs, 10ores: watroles/ores returns 500 Internal Server Error, works for other projects - https://phabricator.wikimedia.org/T128871#2088882 (10scfc) a:3scfc The error occurs also for http://tools.wmflabs.org/watroles/project/tools, etc. I'll take a look. [17:34:29] (03PS1) 10Glaisher: Elections: unbreak elections.php [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275001 (https://phabricator.wikimedia.org/T128742) [17:35:03] i want to write a airticle about bangladesh village birds [17:35:58] hi andyrussg [17:36:17] Monir: wrong channel? I think you want #wikipedia-en-help [17:36:44] yes [17:36:58] can u help glaisher [17:37:46] Sorry, no but you can try that channel if you want help with writing an article. [17:38:02] ok. thanks [17:39:32] what, we didn't have a project for birds? [17:40:36] https://twitter.com/ProBirdRights?ref_src=twsrc^google|twcamp^serp|twgr^author [17:42:04] andrewbogott: !:) nice [17:44:37] 6Labs, 10Labs-Infrastructure, 10Tool-Labs, 10ores: watroles/ores returns 500 Internal Server Error, works for other projects - https://phabricator.wikimedia.org/T128871#2088909 (10scfc) Looking at the projects that succeed and those that fail, I assume it's related to their sizes. ``` usage 1:... [17:50:24] 6Labs, 10Tool-Labs, 10WLX-Jury: Figure out a way to support java 1.8 on tool labs (For WLX Jury) - https://phabricator.wikimedia.org/T124903#2088953 (10intracer) With Play 2.5 release https://www.playframework.com/documentation/2.5.x/Highlights25 absense of Java 8 means staying 2 major releases behind with P... [17:52:39] 6Labs, 10Labs-Infrastructure, 6Operations: Estimate hardware requirements for relevance lab elasticsearch servers - https://phabricator.wikimedia.org/T128433#2088977 (10TJones) I don't have strong opinions on the specific specs (these machines are all studly beyond my ken), but it sounds like something a lit... [17:55:29] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:00:27] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 798785 bytes in 9.952 second response time [18:11:26] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:16:27] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 798688 bytes in 8.683 second response time [18:33:52] 6Labs, 10MediaWiki-ResourceLoader, 10WikiEditor, 10wikitech.wikimedia.org, 13Patch-For-Review: Wikitechwiki has 4xx responses to requests for some static assets inc. poweredby_mediawiki_88x31.png and WikiEditor's button-sprite.svg - https://phabricator.wikimedia.org/T128747#2089228 (10Jdforrester-WMF) p:... [18:34:54] 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Wikitechwiki has 4xx responses to requests for some static assets inc. poweredby_mediawiki_88x31.png and WikiEditor's button-sprite.svg - https://phabricator.wikimedia.org/T128747#2089235 (10Krinkle) [18:35:48] 6Labs, 6Operations, 10wikitech.wikimedia.org, 13Patch-For-Review: Wikitechwiki has 4xx responses to requests for some static assets inc. poweredby_mediawiki_88x31.png and WikiEditor's button-sprite.svg - https://phabricator.wikimedia.org/T128747#2084612 (10Krinkle) Check list: * `public-wiki-rewrites.incl`... [18:36:30] 10Tool-Labs-tools-Other, 6Community-Tech, 7Community-Wishlist-Survey, 7Milestone: Pageview Stats tool - https://phabricator.wikimedia.org/T120497#2089248 (10kaldari) [18:50:45] 6Labs, 10Datasets-General-or-Unknown, 10Dumps-Generation, 6Operations, and 2 others: copy wikitech dumps to dumps server ? - https://phabricator.wikimedia.org/T128680#2089333 (10Dzahn) i merged that but did not see the cron entry yet, needs confirmation on dataset1001 [18:53:28] 6Labs, 10Datasets-General-or-Unknown, 10Dumps-Generation, 6Operations, and 2 others: copy wikitech dumps to dumps server ? - https://phabricator.wikimedia.org/T128680#2089371 (10ArielGlenn) I think you checked it before running puppet. I did a run and it's there now. [18:55:44] 6Labs, 10Datasets-General-or-Unknown, 10Dumps-Generation, 6Operations, and 2 others: copy wikitech dumps to dumps server ? - https://phabricator.wikimedia.org/T128680#2089420 (10Dzahn) odd, i ran puppet even more than once. but there it is now indeed :) [19:45:47] 6Labs, 10Tool-Labs: Server errors of web tools are reported as 404 - https://phabricator.wikimedia.org/T128898#2089602 (10scfc) [19:57:54] 6Labs, 10Labs-Infrastructure, 10Tool-Labs, 10ores: watroles/ores returns 500 Internal Server Error, works for other projects - https://phabricator.wikimedia.org/T128871#2089663 (10scfc) 5Open>3Resolved (TIL: To debug, add `DEBUG = True` to `~tools.watroles/src/watroles/settings.py` and look at `~/uwsgi... [20:05:11] 6Labs, 10Labs-Infrastructure, 10Tool-Labs, 10ores: watroles/ores returns 500 Internal Server Error, works for other projects - https://phabricator.wikimedia.org/T128871#2089688 (10scfc) Pull request at https://github.com/yuvipanda/watroles/pull/1. [20:22:25] !log ganglia created puppet groups bastion/ganglia/installserver, added puppet classes to groups to let us use classes like in prod [20:22:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ganglia/SAL, Master [20:23:22] !log ganglia configured jessie-bastion-01 like bast4001 to debug [20:23:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ganglia/SAL, Master [20:24:34] !log ganglia role keyword not working in labs, not sure if testing is possible anymore [20:24:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ganglia/SAL, Master [20:26:36] !log ganglia reconfigured jessie-bastion-01 with ONLY ganglia aggregator, Error 400 on SERVER: Must pass sites to Class [20:26:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ganglia/SAL, Master [20:34:43] andrewbogott: hello! So today I have learned the hard way that one want to run 'sync' before snapshotting an instance ;D [20:35:24] hashar: I’m guessing that has something to do with write caches? [20:35:36] yup [20:36:35] short story. Instance has a git repo. I git pull then immediately after snapshot the instance [20:37:03] inodes have been written/registered but actual content did not make it [20:37:37] so booting from snapshot result in bunch of files being 0 bytes [20:38:06] and eventually found OS has a doc for it http://docs.openstack.org/openstack-ops/content/snapshots.html :P [20:50:03] 6Labs: Clean up ogvjs-testing labs instance - https://phabricator.wikimedia.org/T128901#2089797 (10brion) (Note the VM instance needs to stay as I have further work on the project on the mobile view!) [21:01:25] !log rcm/cac testlog [21:01:25] rcm/cac is not a valid project. [21:01:29] hm, ok [21:12:48] Luke081515: i don't think the / can be part of a project name [21:13:17] mutante: I just think about creating a seperate SAL at a subpage [21:13:22] not a project with / ;) [21:13:42] ah, ok. yea, so it expects the project name like it exists in wikitech [21:13:49] and the SAL gets autocreated [21:13:55] :-/ [21:14:05] but you can have more projects [21:14:08] they dont hurt [21:14:40] at the moment I have enough quotas left ;) [21:14:45] afaict it doesnt matter if we have 10 instances in one project or 1 instance in 10 projects [21:16:11] btw, I added a link to grafana to the project description page [21:16:36] so user can now choose between grfana and nagf [21:19:42] nice [21:21:36] my second edit was better, the first one destroyed the layout :D [21:30:54] andrewbogott: can I take over the role as project admin for https://wikitech.wikimedia.org/wiki/Nova_Resource:Wmt? [21:31:06] or do I need to file a phab ticket for that [21:32:34] SPF|Cloud: ask the current maintainers [21:32:39] oh, that's andrew :-) [21:33:47] * andrewbogott is trying really hard to remember what ‘wmt’ stands for [21:34:32] wiki monitoring team [21:35:22] Heh, I just do not remember that at all :) But, sure, I’ll add you, since you’re a member already [21:35:31] done [21:35:32] thank you [21:35:50] labs guys are great people [21:35:55] (just like everyone) [21:42:23] 6Labs, 6Operations: labs precise and jessie instance not accessible after provisioning - https://phabricator.wikimedia.org/T117673#2090002 (10Andrew) The initial point of failure in all these cases seems to be that 'hostname -d' fails. That causes a cascade of disaster since things like cert names &c are base... [21:59:27] !log phlogiston Added MBinder as project admin [21:59:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phlogiston/SAL, Master [22:46:48] 6Labs, 10Tool-Labs: Should I report hackery against one of my tools somewhere? - https://phabricator.wikimedia.org/T128918#2090229 (10Rillke) [22:47:13] madhuvishy: I’d like to shut off ores-worker-02 for a little while… do you know if that will cause problems? [22:48:19] hmmm andrewbogott I don't actively admin ores - halfak is the right person to ask. let me see if he's on #research [22:54:08] !log ores migrating ores-worker-02 to labvirt1010 [22:54:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, dummy [23:05:15] 10Tool-Labs-tools-stewardbots: hat-web-tools import for stewardbots - https://phabricator.wikimedia.org/T128743#2090256 (10MarcoAurelio) If you agree, I can copy the files to stewardbots again. I think we have them in a folder with the old toolserver stuff, but not being able to know what exactly is hosted in la... [23:06:43] 10Tool-Labs-tools-stewardbots, 13Patch-For-Review: Make elections.php work again - https://phabricator.wikimedia.org/T128742#2090272 (10MarcoAurelio) I guess we need a cache folder too? [23:17:28] andrewbogott, deployment-tin is using like 24.5 GB [23:17:49] On here it's listed as using 153G [23:18:04] ok, that’s a good test case… mind if I shut it down for a few minutes? [23:18:15] sorry, 23.2 GB [23:18:20] umm [23:18:29] maybe best ask in -releng [23:18:41] that server does automatic deployment of master branches [23:18:53] to beta.wmflabs.org apaches [23:19:40] andrewbogott, this is definitely supposed to be counting the amount of space *used*, not just *available*, right? [23:22:06] (03CR) 10MarcoAurelio: "check zend" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/275001 (https://phabricator.wikimedia.org/T128742) (owner: 10Glaisher) [23:22:58] There are other instances listed as having a much larger disk footprint than they're actually using, e.g. deployment-db1 [23:23:51] Krenair: you’re correct. But it’s copy-on-write, not delete-on-unwrite, if that makes sense. [23:24:25] So empty space on a new instance isn’t consumed in the file system. But if you create a million files and then delete them one by one... [23:24:43] shrinking the image in that case is not such an easy problem as I understand it. [23:25:05] 10Tool-Labs-tools-stewardbots, 10Continuous-Integration-Config: Implement jenkins tests on labs/tools/stewardbots - https://phabricator.wikimedia.org/T128503#2090346 (10MarcoAurelio) p:5Triage>3Normal Not sure I'll be able to get this done. [23:25:07] I’m right now re-compressing deployment-tin, we’ll see what pans out [23:25:08] right, that makes sense [23:27:05] 79G 088a0575-c09b-4c42-88a5-4ef57d8705c0 deployment-restbase02 deployment-prep mobrovac [23:27:35] yet it appears to have <30GB actually available? [23:27:47] it has /dev/vda3 but not /dev/vda1 or /dev/vda2 [23:31:54] !log ganglia tried setting $sites via hiera page, via hieradata/labs, then run into Reading data from Ganglia failed: TypeError - that's from realm.pp:82 and $dnsconfig = hiera_hash('labsdnsconfig', {}) [23:31:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ganglia/SAL, Master [23:32:43] !log ganglia so either "must pass $sites" or "Reading data ..failed" .. [23:32:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ganglia/SAL, Master [23:34:55] 6Labs, 10Tool-Labs, 10CirrusSearch, 6Discovery: "Search Tool Labs Documentation" doesn't work - https://phabricator.wikimedia.org/T128758#2090379 (10bd808) p:5Triage>3High Not being able to search to Tool Labs docs on wikitech is a fairly big deal. [23:35:12] Krenair: no dice, recompressing only reduced deployment-tin by 2G [23:35:22] Maybe there’s something about fragmentation? [23:42:21] !log ganglia will try switching to puppet::self - cant get it to work [23:42:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ganglia/SAL, Master [23:48:51] !log ganglia removed labs hieradata again, configured instance to not use ganglia class anymore at all - error does NOT go away - deleting instance [23:48:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ganglia/SAL, Master [23:55:22] !log ganglia created jessie-bastion-02, configured with puppet::self, -> invoke-rc.d: initscript puppetmaster, action "start" failed. [23:55:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ganglia/SAL, Master [23:56:00] !log ganglia - Error: Could not send report: Connection refused - connect(2) for "localhost" port 8140 [23:56:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ganglia/SAL, Master [23:56:58] !log ganglia - giving up for now [23:57:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ganglia/SAL, Master [23:59:57] 6Labs, 10Labs-Infrastructure: puppet::self broken - https://phabricator.wikimedia.org/T128930#2090480 (10Dzahn)