ZerotoDocker is a project that allows anyone with a Docker host to run a single node of any Netflix OSS technology with a single command. The portability of Docker allows us to run the tools locally or in different cloud environments such as AWS or Azure. However, it is important to keep in mind that some of the Netflix OSS tools work only in AWS. In these cases, although we could start a Docker container running the application in other environments such as Azure, the tools won’t be able to provide the expected functionality.
If you are not familiar with Docker and you would like to read more about it, you can check out this blog post.
Available Docker Images
Netflix OSS provides Docker images for the following tools:
- A Karyon based Hello World Example
- A Zuul Proxy used to proxy to the Karyon service
- Security Monkey
- Exhibitor managed Zookeeper
It is important to keep in mind that these images are not intended to be used in production environments.
Additionally, as we mentioned before, some of the Netflix OSS services corresponding to the images offer functionalities associated exclusively with AWS:
- Atlas: According to the Atlas wiki in the Zero to Docker repository, it appears that the Atlas image requires the AWS APIs in order to work.
- Asgard: It offers a web interface for application deployments and cloud management in Amazon Web Services (AWS)
- Edda: It polls AWS resources via AWS APIs and records the results.
- Security Monkey: It monitors policy changes and alerts on insecure configurations in an AWS account.
Evaluating Netflix OSS
In this section we show how you can test Genie and the “Hello Netflix OSS” sample application in a Docker environment. This sample application is based on Karyon and interacts with Eureka and Zuul.
As we mentioned before, these images can be run in different environments. In our case, we will test them in Azure.
If you would like to know how to set up an Azure VM with Docker, please take a look at these posts:
Running Genie on Docker
This section describes the steps to set up Genie 2.2.1 using the Docker image. If you’re looking for a different version, please see the list of releases here. The steps described in this document are based on the instructions provided here.
Please, consider that the Docker image we will use is not considered production ready.
This section describes how to set up and configure the containers required to run the example.
The first step is to set up MySQL. In order to start a new container running the MySQL image, we need to run the following command:
docker run –name mysql-genie -e MYSQL_ROOT_PASSWORD=genie -e MYSQL_DATABASE=genie -d mysql:5.6.21
If you don’t have the MySQL image in your host, it will be downloaded. Otherwise, the container will start using the existing image:
The previous command will start a container named “mysql-genie”. We’ll use that name later to reference this container from Genie in order to establish a connection.
To verify if MySQL is running properly, we can do 2 things:
- Run the “docker ps” command to check that the container is running
- Access the MySQL container
Set up Hadoop to Run Example
We just need to run the “sequenceiq/hadoop-docker” image. In this case, we will run the command in interactive mode to be able to configure our Hadoop container and verify that everything is working as expected.
docker run –name hadoop-genie -it -p 10020:10020 -p 19888:19888 -p 211:8088 sequenceiq/hadoop-docker:2.6.0 /etc/bootstrap.sh -bash
Since we already have an endpoint configured for port 211 in our Azure VM, we included the port mapping “211:8088” to be able to access the Hadoop Resource manager.
Once we have Hadoop running, we will modify the /etc/hosts file. We will add “hadoop-genie” (the container name) after the container id to the first line (space separated). This will allow the daemons to resolve each other when a job is submitted in the future from the Genie node by container name.
So, we will run the following command to start editing the hosts file.
After editing the file, it should look as follows:
172.17.0.7 83893db7d234 hadoop-genie
::1 localhost ip6-localhost ip6-loopback
Finally, we will start the Job History Server:
/usr/local/hadoop/sbin/mr-jobhistory-daemon.sh start historyserver
Running the “jps” command, we should see something like this:
We will leave Hadoop running in the current SSH client and open a new one to start working with Genie.
Run the Genie Container
Once we have opened a new SSH client, accessed our VM and configured Docker properly, we can run the Genie container.
In our case, we have an endpoint in our Azure VM configured for port 210, so we will run the Genie container mapping public port 210 to container port 8080, which is the default Genie port.
docker run -p 210:8080 –name genie –link mysql-genie:mysql-genie –link hadoop-genie:hadoop-genie -d netflixoss/genie:2.2.1
Once the Genie container is running, we can verify if everything is working properly.
First, we will check the connection with the MySQL container from Genie:
- Access the MySQL container by running:
docker exec -it mysql-genie mysql -pgenie
- Run the “use genie” command
- Finally, check the genie database tables by running:
To exit the MySQL container, we need to run “exit”.
Once we have verified the connection with the database, we can access the Genie UI from a browser. We should be able to access Genie by accessing our VM URL and providing port 210. In our case, the URL is:
As you can see, there are no clusters or jobs.
In order to finish our verification, we will check that the connection with Hadoop is working. To do this, we will access the Genie container by running:
docker exec -it genie /bin/bash
Then, we will ping the Hadoop container. We should see information about packages that have been sent and received successfully.
To stop the ping command, we will press “Ctrl + C”.
Run the example
We have everything in place, and we are ready to run the example. The example configures Genie with the Hadoop configuration information for the Hadoop container mentioned earlier as well as two commands (Hadoop and Pig). Then, it will launch a Hadoop MR job which is the example provided by Hadoop: ”hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output ‘dfs[a-z.]+’”.
We are already in the Genie container, so we can start to run the example.
First, we’ll execute the setup script to register the Hadoop cluster and Hadoop / Pig commands.
In order to verify that everything was registered as expected, we can go to the Genie UI and check the commands and clusters sections:
- Home page:
- Clusters view:
- Commands view:
Additionally, we can verify that “excite.log.bz2” is in HDFS by using:
Finally, if everything is OK, we can run the example job.
Once the Job is started, we will see it in the Genie UI:
If we access the Jobs section, we should see it there too:
We can also see the output of the Job by accessing:
Here, we can check the different logs.
When the job finishes executing, the console should resemble the following:
That’s it! We ran Genie and submitted a job to a Hadoop cluster. Additionally, we were able to check the different logs corresponding to the job.
Running the “Hello Netflix OSS” sample on Docker
This section describes how to run the “Hello Netflix OSS” sample image in combination with Eureka and Zuul, running the corresponding Docker images in an Azure VM.
Please take into account that the Docker images we will use are not considered production-ready.
The first thing we will do is to start a container running the Eureka image. Since we have an endpoint for port 212 in our Azure VM, we will map it to container port 8080 since Eureka is accessible through that port.
docker run -d -p 212:8080 –name eureka netflixoss/eureka:1.1.147
After running the image, we should be able to access the Eureka page through the following URL:
In our case, the URL is:
Hello Netflix OSS sample application
Once Eureka is running, we can start the sample image:
docker run -d -p 213:8080 -p 214:8077 –name hello-netflix-oss –link eureka:eureka netflixoss/hello-netflix-oss:1.0.27
In this case, we configured the ports to be able to access the application though port 213 and the embedded Karyon admin services console through port 214:
Additionally, we defined a link with the Eureka container to allow Eureka to communicate with the sample application container.
Although the application is already accessible, we will start a container running Zuul to access the sample application through Zuul.
So, we need to start a container running the Zuul image and link it to Eureka.
docker run -e “origin.zuul.client.DeploymentContextBasedVipAddresses=HELLO-NETFLIX-OSS” -p 210:8080 -d –name zuul –link eureka:eureka netflixoss/zuul:1.0.28
Here, we are making Zuul accessible through port 210 of our Azure VM. If we access the Zuul port, we will see that the sample application is displayed:
At this point, if we check the Eureka application, it will show both applications: Zuul and the sample application.
In this scenario we were able to run an application based on Karyon, establish a communication with Eureka and access the application through Zuul.