Ariane Robineau
Head of 3D
In charge of 3D
Render farm
Launch easily your 3D renderings on Render, our dedicated platform.
Try for free

White Paper - 3D Animation and VFX on Qarnot

September 7, 2021 - 3D

Image rendered by Iconem on Qarnot

How to use Qarnot

This section pinpoints the best practices to use Qarnot. There are several ways to send your renders on Qarnot:

  • it is possible to use Render, a web interface that does not require any computing skills. A tutorial about Render can be found here.
  • Qarnot provides a Python SDK (mostly used by the studios we’re working with) that makes possible to integrate Qarnot to the studio’s pipeline and automate the submission of rendering tasks.

Before starting to use Qarnot it is needed to create an account here. With this account, access is given to all software supported by Qarnot, and to about sixty hours of free rendering.

Upload data

Prepare your data (split data between several buckets)

It is possible to use several input bucket on Qarnot doing the following command: task.resources.append(conn.retrieve_or_create_bucket('BUCKET_1')) task.resources.append(conn.retrieve_or_create_bucket('BUCKET_2')) This will allow data transfer optimization from centralized storage to the Q.Box from which the rendering instance(s) inherits as only the buckets affected by the render will be downloaded. Whenever the data can be splitted between different buckets, it is then recommended to do so. Note: Buckets on Qarnot are mounted (by default) in /job in the Docker container that runs the render.

Data upload

Qarnot's centralized object storage service uses the S3 protocol, which means that it is possible to use any S3 compatible tool to upload data on Qarnot. For 3D and VFX studios, it is recommended to configure and use one of these tools: S3Browser, Cyberduck, s3cmdline, … More details on how to configure these tools can be found at the end of this link.

Launch a simple render using the Python SDK

The Qarnot Python SDK allows to start renders using only a few lines of Python code, it can be installed by following these instructions. The minimal way to launch a Blender 2.92 render on Qarnot for example is to write and execute the following script: #!/usr/bin/env python3 import qarnot conn = qarnot.Connection(client_token='YOUR_TOKEN') task = conn.create_task('RENDER_NAME', 'blender-2.92', 1) task.resources.append(conn.retrieve_or_create_bucket('INPUT_BUCKET')) task.results = conn.create_bucket('BUCKET_OUT') task.constants['BLEND_FILE'] = 'BLENDER_FILE' task.submit() Note: You can find your token in your account.

Automatize renders on Qarnot

Python SDK

The Python SDK documentation is available here. A few tricks can be useful to your renders:

  • It is possible to start several instances using: task = conn.create_task('RENDER_NAME', 'blender-2.92', NUMBER_OF_INSTANCES_HERE) or: task = conn.create_task('RENDER_NAME', 'blender-2.92', 0) task.advanced_range = 'START-END'
  • An instance is referenced through the $INSTANCE_ID environment variable. This environment variable will be passed to the running container. It can be used to refer to frame IDs for example, so that the rendered frames are automatically dispatched to all instances involved in the computation.
  • it is possible to add environment variables and reusing them in render jobs on Qarnot by writing: task.constants['ENV_VAR'] = 'VALUE_HERE' These variables will be passed to the running container.

Monitoring

Monitoring your rendering tasks can be done using several methods:

  • Render: It is possible to use the Render interface to get more details about the instances information and the render itself. By accessing the task’s details, it is possible to get the renders detailed logs, but also the characteristics of the render.
  • Python SDK: You can get logs in real time through your own terminal (or in log files) writing the following lines: sys.stdout.write(task.fresh_stdout()) sys.stderr.write(task.fresh_stderr())

Special cases

Licenses

Qarnot provides on-demand licencing with adapted rates (depending on the software licence cost) for all available software. Please note that rendering with your own licenses with a BYOL method (Bring Your Own Licence) method is possible and secure, contact us to know more.

Custom versions of software

When rendering with Qarnot it is possible to use the software combinations available (they are all listed in the profile list here. If something is missing, please contact us), but it is also possible to use your own custom versions of software. The best method is to bring your own Docker image. This can be done by putting your own Docker images on the Docker Hub or on a private registry. Access and use of these images is then possible thanks to the Qarnot docker-batch profile. In certain cases, it is possible to put your software or needed files (ocio, shaders,...) in a bucket that will be mounted on the running container. The files can then be accessed with the /job prefix where the bucket is mounted by default. Please note that plugins are compatible with Qarnot. Their adaptation can be done using one of the methods described above. If using one of these methods isn’t possible nor convenient, please contact us.

How Qarnot works

This section explains how Qarnot works and what happens when a user uploads data to Qarnot and sends rendering tasks. There are three main physical sites concerned by the transfers:

  • The studio itself.
  • Traditional data centers where Qarnot has a few replicated servers that host:
    • the API (api.qarnot.com) used to send rendering tasks on Qarnot infrastructure.
    • a centralized and redundant object storage (storage.qarnot.com).
    • Qarnot’s apps (render, console) and websites.
  • Qarnot's sites, where heaters and boilers are installed.

What happens when a rendering task is sent to Qarnot is:

  1. The user’s upload of the dataset to the central storage system (storage.qarnot.com)
  2. When a rendering task is submitted, the data relevant for the render is sent to a Q.Box on a Qarnot site that needs heating. A Q.Box is a server temporarily storing the data for the computation.
  3. Once the data is downloaded on the Q.Box, Docker containers are started on the instances inheriting from the involved Q.Box, and the data is mounted on the containers. When the task is finished (though it’s possible to get output data during the task), output data is uploaded from the Q.Box to the centralized storage and is accessible to the studio.

Share on networks