Tensorflow on Qarnot Cloud - documentation

Introduction

TensorFlow is a popular end-to-end open source platform for developing and deploying machine learning applications using the many tools and libraries that comprise it. Using Qarnot for training and getting predictions for your TensorFlow model is a simple task. Here is a quick step by step walkthrough to guide you through the different steps of how to do that so follow along!

Version

If you are interested in another version, please send us an email at qlab@qarnot.com.

Prerequisites

Before starting a calculation with the Python SDK, a few steps are required:

• Retrieve the authentication token (here)
• Install Qarnot’s Python SDK (here)

Note: in addition to the Python SDK, Qarnot provides C# and Node.js SDKs and a Command Line.

Test Case

This test case will showcase how to train a dogs and cats image classifier based on a pretrained model and make predictions using Qarnot’s Pools. The necessary input files needed for this tutorial can be downloaded here.

Once you have downloaded everything, your working directory should look like this:

• buckets
  • dogcats-small: images used for labeling
  • learn
  • animals: images for training
  • pretrained_model: the pretrained model
  • scripts: scripts that will run on Qarnot
• qlabel_pool.py: make predictions in pools (code found below)
   
• qlearn.py: train the dog & cat image classifier (code found below)
• sync_bucket.py: upload the necessary files to Qarnot (code found below)

Make sure to add the scripts in their respective places as they show up in the article.

Launching the test case

Once you have set up your working environment, you can start with the first step: training the dog & cat image classifier. Be sure to copy your authentication token in the script (instead of <<<MY_SECRET_TOKEN>>>) to be able to launch the task on Qarnot.

Training the image classifier 

Next you have to transfer your local data to buckets on Qarnot. To do so, copy the following code in a python script and run python3 sync_bucket.py from your terminal.

Once the transfer is done, you should see your newly created buckets on Tasq and you can inspect the images you just uploaded ! Below is an example of an input image of a cat.

You are now ready to train your dog and cat classifier on Qarnot by copying the following code in a python script and running python3 qlearn.py & from your terminal.

Training results

At any given time, you can monitor the status of your task on Tasq. Once training is done, the task should change into Success state with a green color.

You can then check your newly trained classifier in the output bucket tensorflow-out-model.

Labeling the cat and dog images

Now that you have retrained the model for dog and cat image classification, it’s time to do some labeling. To do so, it will be done in a pool. In order to launch the classification task, copy the following code in a python script and run python3 qlabel_pool.py & from your terminal.

The script above will launch a pool that provisions 10 servers and run a task with 100 computation instances, each instance will label one of the 100 images you transferred in the tensorflow-in-dogscats-small bucket. Once the task is running, you can view its progress from your Console and see the status of each instance in the instance visualizer on the left.

Labeling results

Once the task is finished, you can inspect the results in the tensorflow-out-sorted-pool bucket in Console and look at the different images your model labeled.

Wrapping up

That’s it! If you have any questions or if you are interested in running this use case and others using GPUs, please contact qlab@qarnot.com and we will help you with pleasure!