TensorFlow is a popular end-to-end open source platform for developing and deploying machine learning applications using the many tools and libraries that comprise it. Using Qarnot for training and getting predictions for your TensorFlow model is a simple task. Here is a quick step by step walkthrough to guide you through the different steps of how to do that so follow along!
If you are interested in another version, please send us an email at email@example.com.
Before starting a calculation with the Python SDK, a few steps are required:
This test case will showcase how to train a dogs and cats image classifier based on a pretrained model and make predictions using Qarnot’s Pools. The necessary input files needed for this tutorial can be downloaded here.
Once you have downloaded everything, your working directory should look like this:
dogcats-small: images used for labeling
animals: images for training
pretrained_model: the pretrained model
scripts: scripts that will run on Qarnot
qlabel_pool.py: make predictions in pools (code found below)
qlearn.py: train the dog & cat image classifier (code found below)
sync_bucket.py: upload the necessary files to Qarnot (code found below)
Make sure to add the scripts in their respective places as they show up in the article.
Launching the test case
Once you have set up your working environment, you can start with the first step: training the dog & cat image classifier. Be sure to copy your authentication token in the script (instead of
<<<MY_SECRET_TOKEN>>>) to be able to launch the task on Qarnot.
Training the image classifier
Next you have to transfer your local data to buckets on Qarnot. To do so, copy the following code in a python script and run
python3 sync_bucket.py from your terminal.
Once the transfer is done, you should see your newly created buckets on your Console and you can inspect the images you just uploaded ! Below is an example of an input image of a cat.
You are now ready to train your dog and cat classifier on Qarnot by copying the following code in a python script and running
python3 qlearn.py & from your terminal.
At any given time, you can monitor the status of your task on your Console. Once training is done, the task should change into Success state with a green color.
You can then check your newly trained classifier in the output bucket
Labeling the cat and dog images
Now that you have retrained the model for dog and cat image classification, it’s time to do some labeling. To do so, it will be done in a pool. In order to launch the classification task, copy the following code in a python script and run
python3 qlabel_pool.py & from your terminal.
The script above will launch a pool that provisions 10 servers and run a task with 100 computation instances, each instance will label one of the 100 images you transferred in the
tensorflow-in-dogscats-small bucket. Once the task is running, you can view its progress from your Console and see the status of each instance in the instance visualizer on the left.
Once the task is finished, you can inspect the results in the
tensorflow-out-sorted-pool bucket in Console and look at the different images your model labeled.
That’s it! If you have any questions or if you are interested in running this use case and others using GPUs, please contact firstname.lastname@example.org and we will help you with pleasure!