Encryption at Rest

New in version 2019.1.1: Scylla Enterprise

Scylla Enterprise protects your sensitive data with data-at-rest encryption. It protects the privacy of your user’s data, reduces the risk of data breaches and helps meet regulatory requirements. In particular, it provides an additional level of protection for your data persisted in storage or its backups.

When Scylla Enterprise encryption at rest is used together with encryption in transit (Node to Node and Client to Node) you benefit from end to end data encryption.

Encryption at Rest is available for Scylla Enterprise customers version 2019.1.1 and later.

About Encryption at Rest

The following can be encrypted:

  • Scylla persistent tables (SSTables)
  • System level data, such as:
    • Commit logs
    • Batches
    • hints logs
    • KMIP Password (part of scylla.yaml)

Encryption at rest works at table level granularity, so you can choose to encrypt only sensitive tables. For both system and table data you can use different algorithms that are supported by OpenSSL in a file block encryption scheme.

Note

SSTables of a particular table can have different encryption keys, use different encryption algorithms, or not be encrypted at all - at the same time.

When is Data Encrypted?

As SSTables are immutable, tables are encrypted only once, as a result of memtable flush, compaction, or upgrade (with Nodetool upgradesstables).

Once a table is encrypted, all resulting SSTables are encrypted using the most current key and algorithm. When you encrypt an existing table, the new SSTables are encrypted. The old SSTables which existed before the encryption are not updated. These tables are encrypted according to the same actions as described previously.

When is Data Decrypted?

When Scylla reads an encrypted SSTable from disk, it fetches the encryption key’s ID from the SSTable and uses it to extract the key and decrypt the data. When Scylla reads an encrypted system table, it fetches the system table encryption key location from the scylla.yaml file. It locates the key and uses it to extract the key and decrypt the data.

Encryption Keys

Two types of encryption keys are available: System Keys and Table Keys.

System Keys

System keys are used for encrypting system data, such as commit logs, hints and/or other user table keys. The system key is stored as the contents of a local file with a single key which you provide.

Table Keys

Table Keys are used for encrypting SSTables. You can provide your own key or Scylla can make one for you.

Key Providers

When encrypting the system tables or SSTables, you need to state which provider is holding your keys. You can use the following options:

Key Provider Name key_provider Name Description
Local Key Provider LocalFileSystemKeyProviderFactory (default) Stores the key on the same machine as the data.
Replicated Key Provider ReplicatedKeyProviderFactory Stores table keys in a Scylla table where the table itself is encrypted using the system key (not available in 2019.1.1)
KMIP Key Provider KmipKeyProviderFactory External key management server (not available in 2019.1.1)

Cipher Algorithms

The following cipher_algorithims are available for use with Scylla using OpenSSL. Note that the default algorithm (AES/CBC/PKCS5Padding with key strength 128 ) is recommended.

cipher_algorithm secret_key_strength
AES/CBC/PKCS5Padding (default) 128 (default), 192, or 256
AES/ECB/PKCS5Padding 128, 192, or 256
Blowfish/CBC/PKCS5Padding 32-448
DES/CBC/PKCS5Padding 56
DESede/CBC/PKCS5Padding 112 or 168
RC2/CBC/PKCS5Padding 40-128

About Local Key Storage

Local keys are used for encrypting user data such as SSTables. Currently, this is the only option available for user data and as such is the default key storage manager. With local key storage, keys are stored locally on disk in a text file. The location of this file is specified in the scylla.yaml.

Caution

Care should be taken so that no unauthorized person can access the key data from the file system. Make sure that the owner of this file is the scylla user and that the file is not readable by other users, not accessible by other roles.

You should also consider keeping the key directory on a network drive (using TLS for the file sharing) to avoid having keys and data on the same storage media, in case your storage is stolen or discarded.

Encrypt a Single Table

This procedure demonstrates how to encrypt a new table. You do not have to make a key, as Scylla will create one for you in the secret_key_file path. If you do want to make your own key, use the procedure in Create Encryption Keys and skip to step 3.

Procedure

  1. By default, the encryption key is located in the /etc/scylla/ directory and the file is named data_encryption_keys. If you want to save the key in a different directory, create one. This example will create encryption keys in different directory (/etc/scylla/encryption_keys, for example) which ensures that the owner of this directory is scylla and not another user.
sudo -u scylla mkdir /etc/scylla/encryption_keys
  1. Create the keyspace if it doesn’t exist.
  2. Create the table using the CREATE TABLE CQL statement, adding any additional options. To encrypt the table, use the options for encryption below, remembering to set the secret_key_file <path> to the same directory you created in step 1.
CREATE TABLE <keyspace>.<table_name> (...<columns>...) WITH
  scylla_encryption_options = {
    'cipher_algorithm' : <hash>,
    'secret_key_strength' : <len>,
    'key_provider': <provider>,
    'secret_key_file': <path>
  }
;

Where:

  • cipher_algoritm - The hashing algorithm which is to be used to create the key. See Cipher Algorithms for more information.
  • secret_key_strength - The length of the key in bytes. This is determined by the cipher you choose. See Cipher Algorithms for more information.
  • key_provider is the name or type of key provider. Refer to Key Providers for more information.
  • secret_key_file - the location that Scylla will store the key it creates (if one does not exist in this location) or the location of the key. By default the location is /etc/scylla/data_encryption_keys.

Example:

Continuing the example from above, this command will instruct Scylla to encrypt the table and will save the key in the location created in step 1.

CREATE TABLE data.atrest (pk text primary key, c0 int) WITH
  scylla_encryption_options = {
    'cipher_algorithm' : 'AES/ECB/PKCS5Padding',
    'secret_key_strength' : 128,
    'key_provider': 'LocalFileSystemKeyProviderFactory',
    'secret_key_file': '/etc/scylla/encryption_keys/data_encryption_keys'
  }
;
  1. From this point, every new SSTable created for the atrest table is encrypted, using the data_encryption_keys key located in /etc/scylla/encryption_keys/. This table will remain encrypted with this key until you either change the key, change the key properties, or disable encryption.
  2. To ensure all SSTables for this table on every node are encrypted, run the Nodetool upgradesstables command. If not, the SSTables remain unencrypted until they are compacted or flushed from MemTables.

For Example:

nodetool upgradesstables data atrest
  1. Your SSTables are encrypted. If you want to change the key at any point, use the Update Encryption Properties of Existing Tables procedure. Always keep your key in a safe location known to you. Do not lose it. See When a Key is Lost.

Update Encryption Properties of Existing Tables

You can encrypt any existing table or use this procedure to change the cipher algorithm, key location or key strength or even disable encryption on a table.

Procedure

  1. Edit the table properties to enable encryption of one table of your choosing. Use the properties explained in Encrypt a Single Table if needed.
ALTER TABLE <keyspace>.<table_name> (...<columns>...) WITH
  scylla_encryption_options = {
    'cipher_algorithm' : <hash>,
    'secret_key_strength' : <len>,
    'key_provider': <provider>,
    'secret_key_file': <path>
  }
;

Example:

Continuing the example from above, this command will instruct Scylla to encrypt the table and will save the key in the location created in step 1.

ALTER TABLE data.atrest (pk text primary key, c0 int) WITH
  scylla_encryption_options = {
    'cipher_algorithm' : 'AES/ECB/PKCS5Padding',
    'secret_key_strength' : 192,
    'key_provider': 'LocalFileSystemKeyProviderFactory',
    'secret_key_file': '/etc/scylla/encryption_keys/data_encryption_keys'
  }
;
  1. If you want to make sure that SSTables that existed before this change are also encrypted, you can either upgrade them using the nodetool upgradesstables command or wait until the next compaction. If you decide to wait, Scylla will still be able to read the old unencrypted tables. If you change the key or remove encryption, Scylla will still continue to read the old tables as long as you still have the key. If your data is encrypted and you do not have the key, your data is unreadable.

    • If you decide to upgrade all of your old SSTables run the nodetool upgradesstables command.

      nodetool upgradesstables <keyspace> <table>
      

      For example:

      nodetool upgradesstables ks test
      
    • Repeat this command on all nodes as nodetool runs locally.

  2. If you want to change the key or disable encryption, repeat the Update Encryption Properties of Existing Tables procedure using the examples below as reference.

Examples

To encrypt an existing table named test in keyspace ks:

ALTER TABLE ks.test WITH
  scylla_encryption_options = {
     'cipher_algorithm' : 'AES/ECB/PKCS5Padding',
     'secret_key_strength' : 128,
     'key_provider': 'LocalFileSystemKeyProviderFactory',
     'secret_key_file': '/etc/scylla/data_encryption_keys'
  }
;

To change the cipher algorithm from AES/ECB/PKCS5Padding to DES/CBC/PKCS5Padding and to change the key strength from 128 to 56 on an existing table:

ALTER TABLE ks.test WITH
  scylla_encryption_options = {
     'cipher_algorithm' : 'DES/CBC/PKCS5Padding',
     'secret_key_strength' : 56,
     'key_provider': 'LocalFileSystemKeyProviderFactory',
     'secret_key_file': '/etc/scylla/data_encryption_keys'
  }
;

To disable encryption on an encrypted table named test in keyspace ks:

ALTER TABLE ks.test WITH
   scylla_encryption_options =  { 'key_provider' : 'none’ };

Create Encryption Keys

When encrypting SSTables, you do not have to generate a key. If you do not, Scylla will create one for you.

When encrypting system files however, you need to generate and upload a key.

Use the key generator script

The Key Generator script generates a key in the directory of your choice.

Procedure

  1. Create (if it doesn’t exist) a local directory for storing the key. Make sure that the owner of the directory is “scylla” and not another user. Make sure that the scylla user can read, write, and execute over the parent directory. Following this procedure makes /etc/scylla/resources/ the parent directory of your keys.

To make a secret key, for example:

sudo mkdir -p /etc/scylla/resources/encryption_keys/secret
sudo chown -R scylla:scylla /etc/scylla/resources
sudo chmod -R 700 /etc/scylla/resources

To make a system key, for example:

sudo mkdir -p /etc/scylla/resources/encryption_keys/system
sudo chown -R scylla:scylla /etc/scylla/resources
sudo chmod -R 700 /etc/scylla/resources
  1. Create a key using the local file key generator script making sure that the keyfile owner is scylla and not another user. Run the command:
sudo -u scylla /bin/local_file_key_generator [options] [key-path]

Where:

  • -a,--alg <arg> - the encryption algorithm (i.e. AES, 3DES) you want to use to encrypt the key
  • -c,--append - appends the output to the key file (default is to overwrite)
  • -h,--help - displays the help menu
  • -l,--length <arg> - the length of the encryption key in bits (i.e. 128, 256)
  • -m,--block-mode <arg> - the encryption algorithm block mode (i.e. CBC, EBC)
  • -p,--padding <arg> - the encryption algorithm padding method (i.e. PKCS5)
  • key-path - is the directory you want to place the key into (/etc/scylla/encryption_keys, for example)

For Example:

To create a key using other encryption settings in a different location:

$ sudo -u scylla /bin/local_file_key_generator a AES -m CES -p PKCS5 -l 192 /etc/scylla/encryption_keys/secret_key

To display the key parameters:

$ sudo cat /etc/scylla/encryption_keys/secret_key
AES/CES/PKCS5Padding:192:NtnKtN6NfxQBO3fJFRhEcA==

Once you have created a key, copy the key to each node, using the procedure described in Copy keys to nodes.

Copy keys to nodes

Every key you generate, needs to be copied to the nodes for use in local key providers.

Procedure

  1. Securely copy the key file, using scp or similar, to the same path on all nodes in the cluster. Make sure the key on each target node is moved to the same location as the source directory and that the target directory has the same permissions as the source directory.
  2. Repeat for all nodes in the cluster.

Encrypt System Resources

System encryption is applied to semi-transient on-disk data, such as commit logs, batch logs and hinted handoff data. This ensures that all temporary stored data is encrypted until fully persisted to final SSTable on disk. Once this encryption is enabled, it is used for all system data.

Procedure

  1. Edit the scylla.yaml file - located in /etc/scylla/scylla.yaml and add the following:
system_info_encryption:
   enabled: <true|false>
   key_provider: (optional) <key provider type>


system_key_directory: <path to location of system key>

Where:

  • enabled can be true or false. True is enabled, false is disabled.
  • key_provider is the name or type of key provider. Refer to Key Providers for more information.
  • system_key_directory is the location of the system key you created in Create Encryption Keys.

Example:

system_info_encryption:
   enabled: True
   key_provider: LocalFileSystemKeyProviderFactory


system_key_directory: /etc/scylla/resources/encryption_keys/system
  1. Save the file.
  2. Drain the node with nodetool drain
  3. Restart the scylla-server service.

CentOS, RHEL or Ubuntu 16.04

sudo systemctl restart scylla-server

Ubuntu 14.04 or Debian

sudo service scylla-server restart

Docker (without restarting some-scylla container)

docker exec -it some-scylla supervisorctl restart scylla

When a Key is Lost

It is very important to backup all of your encryption keys in a secure way. Keep a copy of all keys in a secure location. In the event that you do lose a key, your data encrypted with that key will be unreadable.