Upgrade Log
For each upgrade of the network, we will track important changes for node operators' deployments.

2021-08-31 (16:00 UTC) - Parameter Update

    Upgrade height: upgrade is scheduled to happen at epoch 8049.
We expect the Mainnet network to reach this epoch at around 2021-08-31 16:00 UTC.

Proposed Parameter Changes

The Oasis Core 21.2.8 release contains the consensus-params-update-2021-08 upgrade handler which will update the following parameters in the consensus layer:
    staking.params.max_allowances specifies the maximum number of allowances on account can store. It will be set to 16 (default value is 0) to enable support for beneficiary allowances which are required to transfer tokens into a ParaTime.
    staking.params.gas_costs , governance.params.gas_costs and roothash.params.gas_costs specify gas costs for various types of staking, governance and roothash transactions. Gas costs for transactions that were missing gas costs will be added.
    scheduler.params.max_validators is the maximum size of the consensus committee (i.e. the validator set). It will be increased to110 (it was set to 100 previously).

Instructions - Voting

Voting for the upgrade proposal will end at epoch 7876. We expect the Mainnet network to reach this epoch at around 2021-08-24 12:00 UTC.
At this time only entities which have active validator nodes scheduled in the validator set are eligible to vote for governance proposals.
At least 75% of the voting power needs to cast vote on the upgrade proposal for the result to be valid.
At least 90% of the votes need to be yes votes for a proposal to be accepted.
This upgrade will be the first upgrade that will use the new on-chain governance service introduced in the Cobalt Upgrade.
The Oasis Protocol Foundation has submitted an upgrade governance proposal with the following contents:
1
{
2
"v": 1,
3
"handler": "consensus-params-update-2021-08",
4
"target": {
5
"consensus_protocol": {
6
"major": 4
7
},
8
"runtime_host_protocol": {
9
"major": 3
10
},
11
"runtime_committee_protocol": {
12
"major": 2
13
}
14
},
15
"epoch": 8049
16
}
Copied!
To view the proposal yourself, you can run the following command on your online Oasis Node:
1
oasis-node governance list_proposals -a $ADDR | jq
Copied!
where $ADDR represents the path to the internal Oasis Node UNIX socket prefixed with unix: (e.g.unix:/serverdir/node/internal.sock).
The output should look like:
1
[
2
{
3
"id": 1,
4
"submitter": "oasis1qpydpeyjrneq20kh2jz2809lew6d9p64yymutlee",
5
"state": "active",
6
"deposit": "10000000000000",
7
"content": {
8
"upgrade": {
9
"v": 1,
10
"handler": "consensus-params-update-2021-08",
11
"target": {
12
"consensus_protocol": {
13
"major": 4
14
},
15
"runtime_host_protocol": {
16
"major": 3
17
},
18
"runtime_committee_protocol": {
19
"major": 2
20
}
21
},
22
"epoch": 8049
23
}
24
},
25
"created_at": 7708,
26
"closes_at": 7876
27
}
28
]
Copied!
Obtain your entity's nonce and store it in the NONCE variable. You can do that by running:
1
ENTITY_DIR=<PATH-TO-YOUR-ENTITY>
2
ADDRESS=$(oasis-node stake pubkey2address --public_key \
3
$(cat $ENTITY_DIR/entity.json | jq .id -r))
4
NONCE=$(oasis-node stake account nonce --stake.account.address $ADDRESS -a $ADDR)
Copied!
where <PATH-TO-YOUR-ENTITY> is the path to your entity's descriptor, e.g. /serverdir/node/entity/.
To vote for the proposal, use the following command to generate a suitable transaction:
1
oasis-node governance gen_cast_vote \
2
"${TX_FLAGS[@]}" \
3
--vote.proposal.id 1 \
4
--vote yes \
5
--transaction.file tx_cast_vote.json \
6
--transaction.nonce $NONCE \
7
--transaction.fee.gas 2000 \
8
--transaction.fee.amount 2000
Copied!
where TX_FLAGS refer to previously set base and signer flags as described in the Oasis CLI Tools Setup doc.
If you use a Ledger-signer backed entity, you will need to install version 2.3.1 of the Oasis App as described in Installing Oasis App 2.3.1 to Your Ledger. This is needed because the current version of the Oasis App available through Ledger Live, version 1.8.2, doesn't support signing the governance.CastVote transaction type.
To submit the generated transaction, copy tx_cast_vote.json to the online Oasis node and submit it from there:
1
oasis-node consensus submit_tx \
2
-a $ADDR \
3
--transaction.file tx_cast_vote.json
Copied!

Instructions - Before Upgrade System Preparation

    This upgrade will upgrade Oasis Core to version 21.2.8 which:
      Upgrades the BadgerDB database backend from v2 to v3. See BadgerDB v2 to v3 Migration section for required steps to be done before upgrade.
      Has a check that makes sure the file descriptor limit is set to an appropriately high value (at least 50000). While previous versions only warned in case the limit was set too low, this version will refuse to start. Follow the File Descriptor Limit documentation page for details on how to increase the limit on your system.
    Stop your node, replace the old version of Oasis Node with version 21.2.8 and restart your node.
Since Oasis Core 21.2.8 is otherwise compatible with the current consensus layer protocol, you may upgrade your Mainnet node to this version at any time.
For this upgrade, do NOT wipe state.
    Once reaching the designated upgrade epoch, your node will stop and needs to be upgraded to Oasis Core 21.2.8.
      If you upgraded your node to Oasis Core 21.2.8 before the upgrade epoch was reached, you only need to restart your node for the upgrade to proceed.
      Otherwise, you need to upgrade your node to Oasis Core 21.2.8 first and then restart it.
If you use a process manager like systemd or Supervisor, you can configure it to restart the Oasis Node automatically.
The Mainnet's genesis file and the genesis document's hash will remain the same.

BadgerDB v2 to v3 Migration

This upgrade will upgrade Oasis Core to version 21.2.x which includes the new BadgerDB v3.
Since BadgerDB's on-disk format changed in v3, it requires on-disk state migration. The migration process is done automatically and makes the following steps:
    Upon startup, Oasis Node will start migrating all <DATA-DIR>/**/*.badger.db files (Badger v2 files) and start writing Badger v3 DB to files with the .migrate suffix.
    If the migration fails in the middle, Oasis Node will delete all <DATA-DIR>/**/*.badger.db.migrate files the next time it starts and start the migration (of the remaining <DATA-DIR>/**/*.badger.db
    files) again.
    If the migration succeeds, Oasis Node will append the .backup suffix to all <DATA-DIR>/**/*.badger.db files (Badger v2 files) and remove the .migrate suffix from all <DATA-DIR>/**/*.badger.db.migrate files (Badger v3 files).
The BadgerDB v2 to v3 migration is very I/O intensive (both IOPS and throughput) and may take a couple of hours to complete.
To follow its progress, run:
1
shopt -s globstar
2
du -h <DATA-DIR>/**/*.badger.db* | sort -h -r
Copied!
and observe the sizes of various *.badger.db* directories.
For example, if it outputted the following:
1
55G data/tendermint/data/blockstore.badger.db
2
37G data/tendermint/abci-state/mkvs_storage.badger.db.backup
3
32G data/tendermint/abci-state/mkvs_storage.badger.db
4
16G data/tendermint/data/blockstore.badger.db.migration
5
2.9G data/tendermint/data/state.badger.db
6
62M data/persistent-store.badger.db.backup
7
2.1M data/tendermint/abci-state/mkvs_storage.badger.db.backup/checkpoints
8
1.1M data/tendermint/abci-state/mkvs_storage.badger.db.backup/checkpoints/4767601/ca51b06a054b69f2c18b9781ea42f0b00900de199c1937398514331b0d136ec3/chunks
9
1.1M data/tendermint/abci-state/mkvs_storage.badger.db.backup/checkpoints/4767601/ca51b06a054b69f2c18b9781ea42f0b00900de199c1937398514331b0d136ec3
10
1.1M data/tendermint/abci-state/mkvs_storage.badger.db.backup/checkpoints/4767601
11
1.1M data/tendermint/abci-state/mkvs_storage.badger.db.backup/checkpoints/4757601/2ec3a28b1f4a2fcce503f2e80eb5d77b6c0a4d1075e8a14d880ac390338a855e/chunks
12
1.1M data/tendermint/abci-state/mkvs_storage.badger.db.backup/checkpoints/4757601/2ec3a28b1f4a2fcce503f2e80eb5d77b6c0a4d1075e8a14d880ac390338a855e
13
1.1M data/tendermint/abci-state/mkvs_storage.badger.db.backup/checkpoints/4757601
14
36K data/persistent-store.badger.db
15
20K data/tendermint/data/evidence.badger.db
Copied!
then the mkvs_storage.badger.db was already migrated:
    old BadgerDB v2 directory: 37G data/tendermint/abci-state/mkvs_storage.badger.db.backup
    new BadgerDB v3 directory: 32G data/tendermint/abci-state/mkvs_storage.badger.db
and now the blockstore.badger.db is being migrated:
    current BadgerDB v2 directory:
    55G data/tendermint/data/blockstore.badger.db
    new BadgerDB v3 directory: 16G data/tendermint/data/blockstore.badger.db.migration
Note that usually, the new BadgerDB v3 directory is smaller due to less fragmentation.

Extra storage requirements

Your node will thus need to have extra storage space to store both the old and the new BadgerDB files.
To see estimate how much extra space the migration will need, use the du tool:
1
shopt -s globstar
2
du -h <DATA-DIR>/**/*.badger.db | sort -h -r
Copied!
This is an example output from a Mainnet node that uses /srv/oasis/node as the <DATA-DIR>:
1
43G /srv/oasis/node/tendermint/data/blockstore.badger.db
2
28G /srv/oasis/node/tendermint/abci-state/mkvs_storage.badger.db
3
311M /srv/oasis/node/tendermint/data/state.badger.db
4
2.0M /srv/oasis/node/tendermint/abci-state/mkvs_storage.badger.db/checkpoints
5
996K /srv/oasis/node/tendermint/abci-state/mkvs_storage.badger.db/checkpoints/4517601
6
996K /srv/oasis/node/tendermint/abci-state/mkvs_storage.badger.db/checkpoints/4507601
7
992K /srv/oasis/node/tendermint/abci-state/mkvs_storage.badger.db/checkpoints/4517601/ba6218d7be2df31ba6e7201a8585c6435154728e55bbb7df1ffebe683bf60217
8
992K /srv/oasis/node/tendermint/abci-state/mkvs_storage.badger.db/checkpoints/4507601/1e0bf592bb0d99832b13ad91bc32aed018dfc2639e07b93a254a05f6791a19ac
9
984K /srv/oasis/node/tendermint/abci-state/mkvs_storage.badger.db/checkpoints/4517601/ba6218d7be2df31ba6e7201a8585c6435154728e55bbb7df1ffebe683bf60217/chunks
10
984K /srv/oasis/node/tendermint/abci-state/mkvs_storage.badger.db/checkpoints/4507601/1e0bf592bb0d99832b13ad91bc32aed018dfc2639e07b93a254a05f6791a19ac/chunks
11
148K /srv/oasis/node/persistent-store.badger.db
12
36K /srv/oasis/node/tendermint/data/evidence.badger.db
Copied!
After you've confirmed your node is up and running, you can safely delete all the <DATA-DIR>/**/*.badger.db.backup files.

Extra memory requirements

BadgerDB v2 to v3 migration can use a number of Go routines to migrate different database files in parallel.
However, this comes with a memory cost. For larger database files, it might need up to 4 GB of RAM per database, so we recommend lowering the number of Go routines BadgerDB uses during migration (badger.migrate.num_go_routines) if your node has less than 8 GB of RAM.
If your node has less than 8 GB of RAM, set the number of Go routines BadgerDB uses during migration to 2 (default is 8) by adding the following to your node's config.yml:
1
# BadgerDB configuration.
2
badger:
3
migrate:
4
# Set the number of Go routines BadgerDB uses during migration to 2 to lower
5
# the memory pressure during migration (at the expense of a longer migration
6
# time).
7
num_go_routines: 2
Copied!

Installing Oasis App 2.3.1 to Your Ledger

This manual installation procedure is needed until the latest version of the Oasis App, version 2.3.1, becomes available through Ledger Live's Manager.
Unlike Nano S devices, Nano X devices are locked meaning one cannot manual install the latest version of the Oasis App on them. If you use a Nano X device, you will need to temporarily switch to a Nano S device or wait for the new version of the Oasis App to be available through Ledger Live's Manager.

Update Firmware to Version 2.0.0

First, make sure the firmware on your Nano S is up-to-date. At least version 2.0.0 released on May 4, 2021, is required. Follow Ledger's instructions for updating the firmware on your Nano S.

Install Prerequisites for Manual Installation

The manual installation process relies on some tooling that needs to be available on the system:
Most systems should already have Python pre-installed.
1
pip3 install --upgrade ledgerblue
Copied!
You might want to install the packages to a Python virtual environment or via so-called User install (i.e. isolated to the current user).

Download Oasis App 2.3.1

Install Oasis App 2.3.1

Make the downloaded installer executable by running:
1
chmod +x installer_s.sh
Copied!
Connect you Nano S and unlock it. Then execute the installer:
1
./installer_s.sh load
Copied!
Your Nano S will give you the option to either:
    Deny unsafe manager, or
    review the Public Key and Allow unsafe manager.
First review the public key and ensure it matches the Generated random root public key displayed in the terminal.
Then double press the Allow unsafe manager option.
If there is an existing version of the Oasis App installed on your Nano S, you will be prompted with the Uninstall Oasis screen, followed by reviewing the Identifier (it will depend on the version of the Oasis App you have currently installed) and finally confirming deletion on the Confirm action screen.
After the new version of the Oasis App has finished loading, you will be prompted with the Install app Oasis screen, followed by reviewing the Version, Identifier and Code Identifier screens. Ensure the values are as follows:
    Version: 2.3.1
    Identifier (Application full hash on the terminal): E0CB424D3B1C2A0F694BCB6E99C3B37C7685399D59DD12D7CF80AF4A487882B1
    Code Identifier: C17EBE7CD356D01411A02A81C64CDA3E81F193BDA09BEBBD0AEAF75AD7EC35E3
Finally, confirm installation of the new app by double pressing on the Perform installation screen. Your Ledger device will ask for your PIN again.
Installing Oasis App 2.3.1 on a Nano S with the firmware version < 2.0.0 (e.g. 1.6.1) will NOT fail. It will show a different Identifier when installing the app which will NOT match the Application full hash shown on the terminal. However, opening the app will not work and it will "freeze" your Nano S device.

Verify Installation

Open the Oasis App on your Nano S and ensure the Version screen shows version 2.3.1.
Starting the manually installed version of the Oasis App will always show the This app is not genuine screen, followed by Identifier (which should match the Identifier value above) screen. Finally, open the application by double pressing on the Open application screen.
After you've signed your governance.CastVote transaction, you can safely downgrade Oasis App to the latest official version available via Ledger Live, version 1.8.2.
To do that, just open Ledger Live's Manager and it will prompt you to install version 1.8.2.

2021-04-28 (16:00 UTC) - Cobalt Upgrade

    Upgrade height: upgrade is scheduled to happen at epoch 5046.
We expect the Mainnet network to reach this epoch at around 2021-04-28 12:00 UTC.

Instructions - Before upgrade

    Make sure you are running the latest Mainnet-compatible Oasis Node version: 20.12.7.
      If you are running a different 20.12.x Oasis Node version, update to version 20.12.7 before the upgrade.
Version 20.12.7 is backwards compatible with other 20.12.x releases, so upgrade can be performed at any time by stopping the node and replacing the binary.
    To ensure your node will stop at epoch 5046 submit the following upgrade descriptor at any time before the upgrade:
    1
    {
    2
    "name": "mainnet-upgrade-2021-04-28",
    3
    "method": "internal",
    4
    "identifier": "mainnet-upgrade-2021-04-28",
    5
    "epoch": 5046
    6
    }
    Copied!
The upgrade descriptor contains a non-existing upgrade handler and will be used to coordinate the network shutdown, the rest of the upgrade is manual.

Runtime operators

Following section is relevant only for runtime operators that are running storage nodes for active runtimes on the Mainnet.
This upgrade requires a runtime storage node migration to be performed before the upgrade genesis is published. This can be done before the upgrade epoch is reached by stopping all runtime nodes and running the migration.
Backup your node's data directory
To prevent irrecoverable runtime storage data corruption/loss in case of a failed storage migration, backup your node's data directory.
For example, to backup the /serverdir/node directory using the rsync tool, run:
1
rsync -a /serverdir/node/ /serverdir/node-BACKUP/
Copied!
The storage database on all storage nodes needs to be migrated with the following command (using the 21.1.1 binary):
1
oasis-node storage migrate \
2
--datadir <NODE-DATADIR> \
3
--runtime.supported <RUNTIME-ID>
Copied!
After the migration to v5 completes, you will see an output similar to:
1
...
2
- migrating from v4 to v5...
3
- migrating version 24468...
4
- migrated root state-root:195cf7a9a103e7300b2bb4e537cb9935cbebd83e448e67aa55433861a6ad7426 -> state-root:cea105a5d701deab935b94af9e8e0c5af5dcdb61c242bf434da9f11aa8d110ba
5
- migrated root io-root:0850c5a33ee7f45aa92724b7d5f28c9ac9ae8799b88cc5be9773e8aba9526ca7 -> io-root:19713a2b44e1bf868ebee43c36872baa3058870bb890a5e25d1c4cea2622be77
6
- migrated root io-root:477391131f60ac2c22bce9167c7e3783a13d4fb81fddd2d388b4ead6a586fe52 -> io-root:f29f86d491303c5fd7b3572e97cbd65b7487b6b4ac519623afd161cc2e4678b7
Copied!
Take note of the displayed state-root and report it to the Foundation, as it needs to be included in the upgrade's new genesis file. Keep the runtime nodes stopped until the upgrade epoch is reached. At upgrade epoch, upgrade the nodes by following the remaining steps above.

Instructions - Upgrade day

Following steps should be performed on 2021-04-28 only after the network has reached the upgrade epoch and has halted:
Mainnet state at epoch 5046 will be exported and migrated to a 21.1.x compatible genesis file. Upgrade genesis file will be published on the above link soon after reaching the upgrade epoch.
    Verify the provided Cobalt upgrade genesis file by comparing it to network state dump. See instructions in the Handling Network Upgrades guide.
    Replace the old genesis file with the new Cobalt upgrade genesis file.
    Stop your node (if you haven't stopped it already by submitting the upgrade descriptor).
    Replace the old version of Oasis Node with version 21.1.1.
    Start your node.
For more detailed instructions, see the Handling Network Upgrades guide.

Additional notes

Examine the Change Log of the 21.1.1 (and 21.0) releases.
Runtime operators
Note the following configuration change in the 21.0 release.
Storage access policy changes
Due to the changes in the default access policy on storage nodes, at least one of the storage nodes should be configured with theworker.storage.public_rpc.enabled flag set to true.
Otherwise, external runtime clients wont be able to connect to any storage nodes.

2020-11-18 (16:00 UTC) - Mainnet

    Block height when Mainnet Beta network stops: 702000.
We expect the Mainnet Beta network to reach this block height at around 2020-11-18 13:30 UTC.
    Upgrade window:
      Start: 2020-11-18T16:00:00Z.
      End: After nodes representing 2/3+ stake do the upgrade.

Instructions

    Download Oasis Node version 20.12.2, while continuing to run version 20.10.x.
    (optional) Use Oasis Node version 20.12.2 to dump network state at the specified block height. It will connect to the running version 20.10.x node.
    Download the Mainnet genesis file published in the 2020-11-18 release.
    (optional) Verify the provided Mainnet genesis file by comparing it to network state dump. See instructions in the Handling Network Upgrades guide.
    Replace the old Mainnet Beta genesis file with the Mainnet genesis file.
    Stop your node.
    Remove the old 20.10.x version of Oasis Node.
    Update your node's configuration per instructions in Configuration changes below.
    Start your node.
This time, we recommend dumping the network state with the upgraded Oasis Node binary so that the genesis file will be in the canonical form.
The canonical form will make it easier to compare the obtained genesis file with the one provided by us.
For more detailed instructions, see the Handling Network Upgrades guide.

Configuration changes

Since we are upgrading to the Mainnet, we recommend you change your node's configuration and disable pruning of the consensus' state by removing the consensus.tendermint.abci.prune key.
For example, this configuration:
1
...
2
3
# Consensus backend.
4
consensus:
5
# Setting this to true will mean that the node you're deploying will attempt
6
# to register as a validator.
7
validator: true
8
9
# Tendermint backend configuration.
10
tendermint:
11
abci:
12
prune:
13
strategy: keep_n
14
# Keep ~7 days of data since block production is ~1 block every 6 seconds.
15
# (7*24*3600/6 = 100800)
16
num_kept: 100800
17
core:
18
listen_address: tcp://0.0.0.0:26656
19
20
...
Copied!
Becomes:
1
...
2
3
# Consensus backend.
4
consensus:
5
# Setting this to true will mean that the node you're deploying will attempt
6
# to register as a validator.
7
validator: true
8
9
# Tendermint backend configuration.
10
tendermint:
11
core:
12
listen_address: tcp://0.0.0.0:26656
13
14
...
Copied!

2020-10-01 - Mainnet Beta

Instructions

    Stop your node.
    Replace the old genesis file with the Mainnet Beta genesis file published in the 2020-10-01 release.
    Start your node.
You should keep using Oasis Core version 20.10.
For more detailed instructions, see the Handling Network Upgrades guide.

2020-09-22 - Mainnet Dry Run

Instructions

    This is the initial deployment.
Last modified 1mo ago