Building a multi-node Spark cluster on your local machine using vagrant and puppet

I cannot imagine it has been 8 months already since my last post of building a Hadoop cluster.  Anyway, have you ever gone through some pains of setting up a single standalone Spark instance on your local machine at all and dealing with lot of configuration issues these kind of pains?  If yes, it is gonna be more hassle for the multi-node Spark version!  Now no more I can assure that, as with the Spark big data framework has been gaining more popularity, I am going to prepare a Vagrant-Puppet repo for Spark in order to demonstrate how to build one multi-node Spark cluster locally on your work station (In this tutorial, I will build a one Spark master node and two Spark worker node.). The benefit of having a test cluster locally can speed up software development and testing cycles without, needless to say.

For details, please go to:
https://github.com/wwken/Vagrant-Puppet/tree/master/vagrant-spark-cluster

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s