Run a ParaTime Node

This page describes how to run a ParaTime node on the Oasis Network.

These instructions are for setting up a ParaTime node. If you want to run a validator node instead, see the instructions for running a validator node. Similarly, if you want to run a non-validator node instead, see the instructions for running a non-validator node.

If you are looking for some concrete ParaTimes that you can run, see the list of ParaTimes and their parameters.

Oasis Core refers to ParaTimes as runtimes internally, so all configuration options will have runtime in their name.

This guide will cover setting up your ParaTime compute node for the Oasis Network. This guide assumes some basic knowledge on the use of command line tools.

Prerequisites

Before following this guide, make sure you've followed the Prerequisites and Run a Non-validator Node sections and have:

  • Oasis Node binary installed and configured on your system.

  • The chosen top-level /node/ working directory prepared. In addition to etc and data directories, also prepare the following directories:

    • bin: This will store binaries needed by Oasis Node for running the ParaTimes.

    • runtimes: This will store the ParaTime binaries and their corresponding signatures (if they are running in a Trusted Execution Environment).

Feel free to name your working directory as you wish, e.g. /srv/oasis/.

Just make sure to use the correct working directory path in the instructions below.

  • Genesis file copied to /node/etc/genesis.json.

Reading the rest of the validator node setup instructions may also be useful.

Stake Requirements

To be able to register as a ParaTime node on the Oasis Network, you need to have enough tokens staked in your entity's escrow account.

To see the staking requirements for different node roles, use the Oasis CLI tools as described in Common Staking Info.

Currently, both the Mainnet and the Testnet require 100 ROSE/TEST for each role type:

Staking threshold (entity): ROSE 100.0
Staking threshold (node-validator): ROSE 100.0
Staking threshold (node-compute): ROSE 100.0
Staking threshold (node-storage): ROSE 100.0
Staking threshold (node-keymanager): ROSE 100.0
Staking threshold (runtime-compute): ROSE 100.0
Staking threshold (runtime-keymanager): ROSE 100.0

To register a node that is both a validator and a ParaTime node, the entity for which the node is registered would need to satisfy the following:

  • Entity registration staking threshold (currently 100 tokens),

  • Validator node staking threshold (currently 100 tokens),

  • Compute node staking threshold (currently 100 tokens),

  • Storage node staking threshold (currently 100 tokens).

All together, there would need to be at least 400 tokens staked in your entity's escrow account.

To stake the tokens, use our Oasis CLI tools.

If everything was set up correctly, you should see something like below when running Oasis Node Stake Account Info command for your entity's account (this is an example for the Testnet):

Balance:
Total: 0.0 TEST
Available: 0.0 TEST
Active Delegations to this Account:
Total: 20000.0 TEST (20000000000000 shares)
Delegations:
- From: oasis1qrwdwxutpyr9d2m84zh55rzcf99aw0hkts7myvv9
Amount: 20000.0 TEST (20000000000000 shares)
Stake Accumulator:
Claims:
- Name: registry.RegisterEntity
Staking Thresholds:
- Global: entity
- Name: registry.RegisterNode.HG5TB3dbY8gtYBBw/R/cHfPaOpe0vT7W1wu/2rtpk/A=
Staking Thresholds:
- Global: node-compute
Staking Thresholds:
- Global: node-storage
Nonce: 1

The stake requirements may differ from ParaTime to ParaTime and are subject to change in the future.

Register a New Entity or Update Your Entity Registration

If you don't have an entity yet, create a new one by following the Creating Your Entity instructions.

Everything in this section should be done on the localhost as there are sensitive items that will be created.

If you will be running the ParaTime on a new Oasis Node, initialize a new node by following the Initializing a Node instructions.

Then update your entity descriptor by enumerating paths to all your node's descriptors (existing and new ones) in the --entity.node.descriptor flag. For example:

oasis-node registry entity update \
... various signer flags ... \
--entity.node.descriptor /localhost/node1/node_genesis.json,/localhost/node2/node_genesis.json

To confirm all nodes are added to your entity descriptor, run:

cat <PATH-TO-entity.json> | jq

and ensure you see all your nodes' IDs under the "nodes" list.

For example:

{
"v": 2,
"id": "QTg+ZjubD/36egwByAIGC6lMVBKgqo7xnZPgHVoIKzc=",
"nodes": [
"yT1h8/eN0VInQxn3YKcHxvSgGcsjsTSYmdTLZZMBTWI=",
"HG5TB3dbY8gtYBBw/R/cHfPaOpe0vT7W1wu/2rtpk/A="
]
}

Then generate and submit the new/updated entity descriptor via the entity registration transaction by following the Generating Entity Registration Transaction instructions.

Make sure your entity descriptor (i.e. entity.json) is copied to your online server and saved as /node/entity/entity.json to ensure the node's configuration will find it.

You will configure the node to automatically register for the roles it has enabled (i.e. storage and compute roles) via the worker.registration.entity configuration flag.

No manual node registration is necessary.

The ParaTime Identifier and Binary

In order to run a ParaTime node you need to obtain the following pieces of information first, both of these need to come from a trusted source:

  • The ParaTime Identifier is a 256-bit unique identifier of a ParaTime on the Oasis Network. It provides a unique identity to the ParaTime and together with the genesis document's hash serves as a domain separation context for ParaTime transaction and cryptographic commitments. It is usually represented in hexadecimal form, for example: 8000000000000000000000000000000000000000000000000000000000000000

  • The ParaTime Binary contains the executable code that implements the ParaTime itself. It is executed in a sandboxed environment by Oasis Node and its format depends on whether the ParaTime is running in a Trusted Execution Environment (TEE) or not. In the non-TEE case this will be a regular Linux executable (an ELF binary, usually without an extension) and in the TEE case this will be an SGXS binary (usually with a .sgxs extension) that describes a secure enclave together with a detached signature of the binary (usually with a .sigextension).

Like the genesis document, make sure you obtain these from a trusted source.

Compiling the ParaTime Binary from Source Code

In case you decide to build the ParaTime binary from source yourself, make sure that you follow our guidelines for deterministic compilation to ensure that you receive the exact same binary.

When the ParaTime is running in a TEE, a different binary to what is registered in the consensus layer will not work and will be rejected by the network.

Install Oasis Core Runtime Loader

For ParaTimes running inside Intel SGX Trusted Execution Environment, you will need to install the Oasis Core Runtime Loader.

The Oasis Core Runtime Loader binary (oasis-core-runtime-loader) is part of Oasis Core binary releases, so make sure you download the appropriate version specified the Network Parameters page.

Install it to bin subdirectory of your node's working directory, e.g. /node/bin/oasis-core-runtime-loader.

Install ParaTime Binary

For each ParaTime, you need to obtain its binary and install it to runtimes subdirectory of your node's working directory.

For ParaTimes running inside a Trusted Execution Environment, you also need to obtain and install the binary's detached signature to this directory.

For example, for the Cipher ParaTime, you would have to obtain the cipher-paratime.sgxs binary and the cipher-paratime.sig detached signature and install them to /node/runtimes/cipher-paratime.sgxsand /node/runtimes/cipher-paratime.sig.

Install Bubblewrap Sandbox (at least version 0.3.3)

ParaTime compute nodes execute ParaTime binaries inside a sandboxed environment provided by Bubblewrap. In order to install it, please follow these instructions, depending on your distribution:

Ubuntu 18.10+
Fedora
Other Distributions
Ubuntu 18.10+
sudo apt install bubblewrap
Fedora
sudo dnf install bubblewrap
Other Distributions

On other systems, you can download the latest source release provided by the Bubblewrap project and build it yourself.

Make sure you have the necessary development tools installed on your system and the libcap development headers. On Ubuntu, you can install them with:

sudo apt install build-essential libcap-dev

After obtaining the Bubblewrap source tarball, e.g. bubblewrap-0.4.1.tar.xz, run:

tar -xf bubblewrap-0.4.1.tar.xz
cd bubblewrap-0.4.1
./configure --prefix=/usr
make
sudo make install

Note that Oasis Node expects Bubblewrap to be installed under /usr/bin/bwrap by default.

Ensure you have a new enough version by running:

bwrap --version

Ubuntu 18.04 LTS (and earlier) provide overly-old bubblewrap. Follow Other Distributions section on those systems.

Setting up Trusted Execution Environment (TEE)

If the ParaTime is configured to run in a TEE (currently only Intel SGX), you must make sure that your system supports running SGX enclaves. This requires that your hardware has SGX support, that SGX support is enabled and that the additional driver and software components are properly installed and running.

Install SGX Linux Driver

Oasis Core currently only supports the legacy (out-of-tree) Intel SGX Linux driver.

Support for the new Intel SGX support in mainline Linux kernels since version 5.11 is being tracked in oasis-core#3651.

Ubuntu 18.04/16.04

A convenient way to install the SGX Linux driver on Ubuntu 18.04/16.04 systems is to use the Fortanix's APT repository and its DKMS package.

First add Fortanix's APT repository to your system:

echo "deb https://download.fortanix.com/linux/apt xenial main" | sudo tee /etc/apt/sources.list.d/fortanix.list >/dev/null
curl -sSL "https://download.fortanix.com/linux/apt/fortanix.gpg" | sudo -E apt-key add -

And then install the intel-sgx-dkms package:

sudo apt update
sudo apt install intel-sgx-dkms

Some Azure Confidential Computing instances have the Intel SGX DCAP driver pre-installed.

To determine that, run dmesg | grep -i sgx and observe if a line like the following is shown:

[ 4.991649] sgx: intel_sgx: Intel SGX DCAP Driver v1.33

If that is the case, you need to blacklist the Intel SGX DCAP driver's module by running:

echo "blacklist intel_sgx" | sudo tee -a /etc/modprobe.d/blacklist-intel_sgx.conf >/dev/null

Fedora 34/33

A convenient way to install the SGX Linux driver on Fedora 34/33 systems is to use the Oasis-provided Fedora Package for the Legacy Intel SGX Linux Driver.

Other Distributions

Go to Intel SGX Downloads page and find the latest "Intel SGX Linux Release" (not "Intel SGX DCAP Release") and download the "Intel (R) SGX Installers" for your distribution. The package will have driver in the name (e.g., sgx_linux_x64_driver_2.11.0_2d2b795.bin).

Verification

After installing the driver and restarting your system, make sure that the /dev/isgx device exists.

Ensure /dev is NOT Mounted with the noexec Option

Newer Linux distributions usually mount /dev with the noexec mount option. If that is the case, it will prevent the enclave loader from mapping executable pages.

Ensure your /dev (i.e. devtmpfs) is not mounted with the noexec option. To check that, use:

cat /proc/mounts | grep devtmpfs

To temporarily remove the noexec mount option for /dev, run:

sudo mount -o remount,exec /dev

To permanently remove the noexec mount option for /dev, add the following to the system's /etc/fstab file:

devtmpfs /dev devtmpfs defaults,exec 0 0

This is the recommended way to modify mount options for virtual (i.e. API) file system as described in systemd's API File Systems documentation.

Install AESM Service

To allow execution of SGX enclaves, several Architectural Enclaves (AE) are involved (i.e. Launch Enclave, Provisioning Enclave, Provisioning Certificate Enclave, Quoting Enclave, Platform Services Enclaves).

Communication between application-spawned SGX enclaves and Intel-provided Architectural Enclaves is through Application Enclave Service Manager (AESM). AESM runs as a daemon and provides a socket through which applications can facilitate various SGX services such as launch approval, remote attestation quote signing, etc.

Ubuntu 20.04/18.04/16.04

A convenient way to install the AESM service on Ubuntu 20.04/18.04/16.04 systems is to use the Intel's official Intel SGX APT repository.

First add Intel SGX APT repository to your system:

echo "deb https://download.01.org/intel-sgx/sgx_repo/ubuntu $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/intel-sgx.list >/dev/null
curl -sSL "https://download.01.org/intel-sgx/sgx_repo/ubuntu/intel-sgx-deb.key" | sudo -E apt-key add -

And then install the sgx-aesm-service, libsgx-aesm-launch-plugin and libsgx-aesm-epid-plugin packages:

sudo apt update
sudo apt install sgx-aesm-service libsgx-aesm-launch-plugin libsgx-aesm-epid-plugin

The AESM service should be up and running. To confirm that, use:

sudo systemctl status aesmd.service

Docker-enabled System

An easy way to install and run the AESM service on a Docker-enabled system is to use Fortanix's AESM container image.

Executing the following command should (always) pull the latest version of Fortanix's AESM Docker container, map the /dev/isgx device and /var/run/aesmd directory and ensure AESM is running in the background (also automatically started on boot):

docker run \
--pull always \
--detach \
--restart always \
--device /dev/isgx \
--volume /var/run/aesmd:/var/run/aesmd \
--name aesmd \
fortanix/aesmd

Podman-enabled System

Similarly to Docker-enabled systems, an easy way to install and run the AESM service on a Podman-enabled system is to use Fortanix's AESM container image.

First, create the container with:

sudo podman create \
--pull always \
--device /dev/isgx \
--volume /var/run/aesmd:/var/run/aesmd:Z \
--name aesmd \
docker.io/fortanix/aesmd

Then generate the container-aesmd.service systemd unit file for it with:

sudo podman generate systemd --restart-policy=always --time 10 --name aesmd | \
sed "/\[Service\]/a RuntimeDirectory=aesmd" | \
sudo tee /etc/systemd/system/container-aesmd.service

Finally, enable and start the container-aesmd.service with:

sudo systemctl enable container-aesmd.service
sudo systemctl start container-aesmd.service

The AESM service should be up and running. To confirm that, use:

sudo systemctl status container-aesmd.service

To see the logs of the AESM service, use:

sudo podman logs -t -f aesmd

Check SGX Setup

In order to make sure that your SGX setup is working, you can use the sgx-detect tool from the sgxs-tools Rust package.

There are no pre-built packages for it, so you will need to compile it yourself.

sgxs-tools must be compiled with a nightly version of the Rust toolchain since they use the #![feature] macro.

Install Dependencies

Make sure you have the following installed on your system:

On Fedora, you can install all the above with:

sudo dnf install gcc protobuf-compiler pkg-config openssl-devel

On Ubuntu, you can install all the above with:

sudo apt install gcc protobuf-compiler pkg-config libssl-dev

Install Rust Nightly

We follow Rust upstream's recommendation on using rustup to install and manage Rust versions.

rustup cannot be installed alongside a distribution packaged Rust version. You will need to remove it (if it's present) before you can start using rustup.

Install rustup by running:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

If you want to avoid directly executing a shell script fetched the internet, you can also download rustup-init executable for your platform and run it manually. This will run rustup-init which will download and install the latest stable version of Rust on your system.

Install Rust nightly with:

rustup install nightly

Build and Install sgxs-tools

cargo +nightly install sgxs-tools

Run sgx-detect Tool

After the installation completes, run sgx-detect to make sure that everything is set up correctly.

When everything works, you should get output similar to the following (some things depend on hardware features so your output may differ):

Detecting SGX, this may take a minute...
✔ SGX instruction set
✔ CPU support
✔ CPU configuration
✔ Enclave attributes
✔ Enclave Page Cache
SGX features
✔ SGX2 ✔ EXINFO ✔ ENCLV ✔ OVERSUB ✔ KSS
Total EPC size: 92.8MiB
✘ Flexible launch control
✔ CPU support
? CPU configuration
✘ Able to launch production mode enclave
✔ SGX system software
✔ SGX kernel device (/dev/isgx)
✘ libsgx_enclave_common
✔ AESM service
✔ Able to launch enclaves
✔ Debug mode
✘ Production mode
✔ Production mode (Intel whitelisted)

The important part is the checkbox under Able to launch enclaves in both Debug mode and Production mode (Intel whitelisted).

In case you encounter errors, see the list of common SGX installation issues for help.

Configuration

In order to configure the node create the /node/etc/config.yml file with the following content:

datadir: /node/data
log:
level:
default: info
tendermint: info
tendermint/context: error
format: JSON
genesis:
file: /node/etc/genesis.json
consensus:
tendermint:
core:
listen_address: tcp://0.0.0.0:26656
# The external IP that is used when registering this node to the network.
# NOTE: If you are using the Sentry node setup, this option should be
# omitted.
external_address: tcp://{{ external_address }}:26656
p2p:
# List of seed nodes to connect to.
# NOTE: You can add additional seed nodes to this list if you want.
seed:
- "{{ seed_node_address }}"
runtime:
supported:
# List of ParaTimes that the node should support.
- "{{ runtime_id }}"
paths:
# Paths to ParaTime binaries for all of the supported ParaTimes.
"{{ runtime_id }}": {{ runtime_bin_path }}
# The following section is required for ParaTimes which are running inside the
# Intel SGX Trusted Execution Environment.
sgx:
loader: /node/bin/oasis-core-runtime-loader
signatures:
# Paths to ParaTime signatures.
"{{ runtime_id }}": {{ runtime_sig_path }}
worker:
registration:
# In order for the node to register itself, the entity.json of the entity
# used to provision the node must be available on the node.
entity: /node/entity/entity.json
storage:
enabled: true
compute:
enabled: true
client:
# External gRPC configuration.
port: 30001
addresses:
# The external IP that is used when registering this node to the network.
- "{{ external_address }}:30001"
p2p:
# External P2P configuration.
enabled: true
port: 30002
addresses:
# The external IP that is used when registering this node to the network.
- "{{ external_address }}:30002"
# The following section is required for ParaTimes which are running inside the
# Intel SGX Trusted Execution Environment.
ias:
proxy:
address:
# List of IAS proxies to connect to.
# NOTE: You can add additional IAS proxies to this list if you want.
- "{{ ias_proxy_address }}"

Before using this configuration you should collect the following information to replace the variables present in the configuration file:

Make sure theworker.client.port (default: 30001) and worker.p2p.port (default: 30002) ports are exposed and publicly accessible on the internet (forTCPtraffic).

Starting the Oasis Node

You can start the node by running the following command:

oasis-node --config /node/etc/config.yml

Checking Node Status

To ensure that your node is properly connected with the network, you can run the following command after the node has started:

oasis-node control status -a unix:/node/data/internal.sock

Troubleshooting

See the general troubleshooting section, before proceeding with ParaTime node-specific troubleshooting.

Too Old Bubblewrap Version

Double check your installed bubblewrap version, and ensure is at least of version 0.3.3. Previous versions of this guide did not explicitly mention the required version. For details see the Install Bubblewrap Sandbox section.

Missing libsgx-aesm-epid-plugin

If you are encountering the following error message in your node's logs:

failed to initialize TEE: error while getting quote info from AESMD: aesm: error 30

Ensure you have all required SGX driver libraries installed as listed in Install SGX Linux Driver section. Previous versions of this guide were missing the libsgx-aesm-epid-plugin.

Stake Requirement

Double check your node entity satisfies the staking requirements for a ParaTime node. For details see the Stake Requirements section.

Runtime Worker Ports

Make sure the worker.client.port (default: 30001) and worker.p2p.port (default: 30002) ports are exposed and publicly accessible on the internet (for TCP traffic). Previous versions of this guide did not explicitly mention this requirement.

Unable to Launch Enclaves

If running sgx-detect --verbose reports:

🕮 SGX system software > Able to launch enclaves > Debug mode
The enclave could not be launched.
debug: failed to load report enclave
debug: cause: failed to load report enclave
debug: cause: Failed to map enclave into memory.
debug: cause: Operation not permitted (os error 1)

Ensure your system's /dev is NOT mounted with the noexec mount option.