Tech Specs
This tutorial will walk you through OpSep's hybrid encryption so that you understand the trustless nature of the protocol. In production, you will use client libraries that abstract away all of this complexity (see example at the bottom of this page).
Table of Contents
- Upgrade OpenSSL
- Grab an RSA Public Key
- Generate a Random DEK Locally
- Encrypt Your Secret Locally Using Your DEK
- Encrypt Your DEK with OpSep’s KEK
- Throw Away the DEK
- Recover Your DEK Using OpSep’s API
- Decrypt Your Ciphertext Locally to Recover Your Secret
- Rate Limiting
- Decryption Audit Log
Before we get started, it's important to understand the terms Data Encryption Key (DEK) and Key Encryption Key (KEK). We use DEKs to encrypt data, and a KEK to encrypt DEKs. We'll use a unique DEK for each record we encrypt, and the same KEK for each DEK. Your DEKs use symmetric encryption, because it is highly performant. Your KEK uses asymmetric encryption, so we can encrypt unlimited records using the same RSA pubkey (and locally without hitting an external server). Your KEK will be guarded by an OpSep server controlled by your security team (we recommend hardening this with an HSM).
Upgrade OpenSSL
This protocol requires openssl
version 1.1
or greater.
You can see what version you're running like this:
$ openssl version
OpenSSL 1.1.1g 21 Apr 2020
If you have an old version of openssl
(or you're actually running LibreSSL
),
you need to upgrade.
Here are instructions
for macOS
and
for Ubuntu.
Grab an RSA Public Key
Grab SecondGuard's testing RSA Public Key. This is the key encryption key (KEK) that you're going to use to encrypt your data encryption keys (DEKs):
$ curl -s https://test.secondguard.com | python3 -c "import sys, json; print(json.load(sys.stdin)['rsaPubKey'].strip())" | awk '{gsub(/\\n/,"\n")}1' > rsa_kek.pub
Do not use this key to protect real data! The private key has intentionally been published. Each OpSep server generates their own key with strict access controls.
Your can see what the public key looks like by running this:
$ cat rsa_kek.pub
-----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA7q4R3soRD2CrjL13OK6Y
SBG8wpjP5sbfkL0QhpJMH87grlR2SS3CUnbYCOONzQiJ3OuKAViy/lMw1KsmG9Nn
hAot2acg1iNyZRY33LR2jwmfFF+2iRp0itPQeOHY6GS8m3WLCMtC/kWUq0Bl5g1P
Ya9JXwSkTTRJunNH0TPk8uqwFeVhpT336M1H6ed105L8a8W3mpSwlwePron7pLf7
wD32m9RT0nNdnHBDQCsUKS/Gdp+saLYWTgj0rpnQCe8f1p3g36Gm0gTzr3X0Adow
8gIPfxO4HU/0cdL+Pw4mpcsWJ4531taRLLGb+a2la2zAUteYcS+8d4Nb8Omkbz39
PylvKP6R1kHElqlF3BnwUp0AdcAvOLdeX8kYUlbKE8xwjHm/KwwleKlcAZDam7hC
Rw72JUQiod0E7My+SiZ3Ij5zKnxZXmAF5BX8T+YSqSzR4Qdp2QU9L9GgAZo/HPBN
wME9v8usjEzrEItSSg3Nn10+J+ygsCqjrCT8CnSvD8wEyDSdO/Jly9DnWJ6B2HJE
Oc4wxWGFTCE0wiQOwC3IPNxFhuWun6/4tsEQcDs5XHaBXIHry5WCiVkjwa2pc95x
iXcfoQWr1A/jLe/MrZyN4yrgDK9mmQxxNzVfLj8S9NPjJMv+K7BKvtOmvoqsf13K
6hYJGkAdR0d99DNFlllRm7cCAwEAAQ==
-----END PUBLIC KEY-----
The same RSA public key can be re-used for every DEK, so this step only has to be performed once. Users typically save this public key locally in their configuration so they don't need to download it each time they want to encrypt a record in their database. Next, we'll use a unique DEK to symmetrically encrypt each record.
Generate a Random DEK Locally
The symmetric data encryption key (DEK) will be used to encrypt your secret locally. Our protocol is designed so that we can never recover your plaintext, even if our systems were compromised.
$ openssl rand -hex 32 > symmetric_dek.txt
Your key should look something like this:
(obviously, don't use this key!)
$ cat symmetric_dek.txt
2c7776dacc5542cceecee5e7867afa08ca251c6d9d91f97856e9a01c6fcd64a6
At 256-bits, this random symmetric key is quite long. An encrypted version of it will later be stored in your database and not something you ever have to remember.
Each record you want to secure in your database will be encrypted with its own unique symmetric DEK. This will be important in step 9 when we discuss Rate Limiting.
At the time you encrypt your data with a unique DEK, your OpSep server will not have access to this DEK. If you later use OpSep to decrypt your data, your OpSep server will be able to reconstruct this symmetric DEK with the data you supply. Recovering your plaintext requires both your ciphertext (which never leaves your server and your OpSep server never sees) as well as this symmetric DEK.
Encrypt Your Secret Locally Using Your DEK
OpSep works with any type of data (string, binary, integer, boolean, etc), but for simplicity we'll protect a string.
We're choosing the secret Attack at dawn
, but you can guard any secret you like:
$ echo "Attack at dawn" | openssl enc -aes-256-cbc -kfile symmetric_dek.txt -pbkdf2 > secret.symm_enc
If you get a Extra parameters given
error, it likely means you're on an outdated version of openssl
or an alternate program like LibreSSL
that is being called instead.
Please go back to the first step (Upgrade OpenSSL) for instructions on how to upgrade.
You can verify this does indeed appear encrypted (your result will be slightly different):
$ cat secret.symm_enc
Salted__??a3??=Hz???l'??f?\?
This is the data you would save to your database (along with symmetric_dek.asymm_enc
in the next step) for later retrieval and decryption.
-aes-256-cbc
instructs openssl
to use AES in Cipher Block Chaining mode.
While asymmetric encryption is slow and has size constraints, this is a very fast symmetric encryption algorithm that can handle large files sizes.
You can encrypt a whole file (to_encrypt.txt
) by first saving it and then passing it into openssl
(no need to pipe it in):
$ openssl enc -aes-256-cbc -kfile symmetric_dek.txt -in to_encrypt.txt -pbkdf2 > secret.symm_enc
Instead of
-kfile symmetric_dek.txt
.
you can enter
-k 2c7776dacc5542cceecee5e7867afa08ca251c6d9d91f97856e9a01c6fcd64a6
if you prefer.
Note that the -k
is lowercase, and replace the hexademical string after it with your symmetric_dek.txt
generated above.
The -pbkdf2
flag is only used to surpress a
*** WARNING : deprecated key derivation used
error that would confuse this exercise.
Without this flag, we get a warning that for (regular) symmetric encryption you want to use a key derivation function.
Since our key is already a massive 256 bits long, there's no reason to suffer the performance hit of a KDF.
However, for this demo we let openssl
perform this unneccsary KDF for simplicity.
You're welcome to get rid of -pbkdf2
throughout these examples if you like; you may then want to add 2> /dev/null
to your openssl
commands to silence this the error.
Encrypt Your DEK with OpSep's KEK
We're using Optimal Asymmetric Encryption Padding with RSA and SHA-256 as our hash function.
Due to the magic of asymmetric cryptography, this takes place locally using the public key (rsa_kek.pub
) that we downloaded earlier.
To start, we'll convert our DEK (symmetric_dek.txt
) to json:
$ echo "{\"key\": \"` cat symmetric_dek.txt `\"}" > symmetric_dek.json
By using a JSON
file, OpSep supports powerful restrictions on future decryption.
For example, we can add an RFC3339 timestamp called deprecate_at
that will mark the key as deprecated at a certain time in the future.
This notation occurs client-side and does not immediately leave your client.
However, if your OpSep server decrypts this file in the future and the deprecate_at
date has passed, then OpSep can refuse to return this deprecated key.
Your OpSep server also supports the client_record_id
and risk_multiplier
(see OpSep server for details).
Now we asymmetrically encrypt symmetric_dek.json
with SecondGuard's public key (rsa_kek.pub
):
$ openssl pkeyutl -in symmetric_dek.json -encrypt -pubin -inkey rsa_kek.pub -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 -pkeyopt rsa_mgf1_md:sha256 | base64 > symmetric_dek.asymm_enc
If you get a parameter setting error
, it likely means you're on an outdated version of openssl
or an alternate program like LibreSSL
that is being called instead.
Please go back to the first step (Upgrade OpenSSL) for instructions on how to upgrade.
You can verify this does indeed appear encrypted:
$ cat symmetric_dek.asymm_enc
This is the data you would save to your database (along with secret.symm_enc
).
Base64 ciphertext is easier to work with than binary data. Encryption performance is not meaningfully affected as this is only the symmetric DEK, which are a small fixed size (unlike your plaintext which could be very large).
While each record you want to secure will require its own unique symmetric DEK, you can encrypt all of them with the same RSA public key.
The corresponding private key (rsa_kek.priv
) is stored in OpSep's HSM, and useage is controlled by a rate-limit policy, access controls, and an intrustion alarm system.
Notice that all the encryption took place locally and is non-interactive, which is great for performance. If you needed to revoke decryption access during a suspected breach (or OpSep were to ever experience downtime) your regular data encryption would continue unaffected.
Throw Away the DEK
If an attacker gets your ciphertext (secret.symm_enc
) and your symmetric DEK (symmetric_dek.txt
& symmetric_dek.json
) they can easily decrypt your ciphertext into plaintext.
By throwing away your symmetric DEK (symmetric_dek.txt
& symmetric_dek.json
), you eliminate this risk and can truly secure your database.
We'll show how to recreate it from your asymmetrically-encrypted symmetric DEK (symmetric_dek.asymm_enc
) in the next step.
For the purpose of this demo we're not bothering to throw away your DEK, because it will make it slightly easier to keep track of what's happening.
If you want to throw away the DEK, enter the following:
$ rm symmetric_dek.txt symmetric_dek.json
Recover Your DEK Using OpSep's API
Let's say some time has passed and you want to decrypt your ciphertext. You make the following query to OpSep's servers to recover your symmetric data encryption key (DEK):
$ curl -s -X POST https://test.secondguard.com/api/v1/decrypt -H 'Content-Type: application/json' -d '{"key_retrieval_ciphertext":"'$(cat symmetric_dek.asymm_enc)'"}' > opsep_api_response.txt
We're using '$(cat ...)'
as a fancy of way passing in the contents of symmetric_dek.asymm_enc
into the json payload that we're POSTing.
Normally, your client library would handle all this complexity for you.
If you prefer, you can manually substitute so that your -d
flag looks something like this:
-d '{"key_retrieval_ciphertext":"jx70uVbQaZeQ...CfTIFI8iqdYX73E"}'
We can parse the symmetric DEK field from that API response:
$ cat opsep_api_response.txt | python3 -c "import sys, json; print(json.load(sys.stdin)['keyRecovered'].strip())" > symmetric_dek_recovered.txt
We can confirm this returned key matches the original by running the following:
(if there were any difference between the files you'd see that displayed here)
$ diff symmetric_dek.txt symmetric_dek_recovered.txt
$
Using python3
is a verbose way to parse the JSON that OpSep's API returns.
If you have jq installed you can instead do:
$ cat opsep_api_response.txt | jq -r '.keyRecovered' > symmetric_dek_recovered.txt
Decrypt Your Ciphertext Locally to Recover Your Secret
Now all that's left to do is locally recover your original secret with your symmetric DEK:
$ openssl enc -d -aes-256-cbc -in secret.symm_enc -kfile symmetric_dek_recovered.txt -pbkdf2
Attack at dawn
While your OpSep server did gain access to your symmetric DEK (symmetric_dek_recovered.txt
), it never saw your ciphertext (secret.symm_enc
) and it has no ability to access your plaintext.
In fact, it doesn't even know what type of data you're protecting (binary, string, integer, boolean, etc), which database you use to store your ciphertext, or even where it is hosted.
Rate Limiting
You may have noticed that the curl
command to OpSep's servers returned the Rate Limiting fields
ratelimitTotal
,
ratelimitRemaining
, and
ratelimitResetsIn
.
Calls to your OpSep server to decrypt your symmetric DEKs are intentionally rate limited.
This means that
even if an attacker or rogue employee has root access to your web server and database,
OpSep's rate-limit policy allows you to configure the amount of data they'll be able to decrypt
before you are notified and they are blocked.
Most of our users set their rate-limit so that their maximum exposure from a breach is < 0.1% of their data. This protection remains even if your attacker has physical control of your server! OpSep also keeps a log of every record your attackers attempt to decrypt. These rules are easy to set and adjust using our web UI with strong multi-factor authentication. We automatically alert you to suspicious decryption requests 24/7 so that you can quickly discover if your servers have been compromised.
You can see these rate limits by running:
$ cat opsep_api_response.txt
Note that the api response will be easier to read if you pipe the output to a json parser like this:
$ cat opsep_api_response.txt | python -m json.tool
Or like this if you have jq installed:
$ cat opsep_api_response.txt | jq
Decryption Audit Log
If a rogue employee or outside attacker gains unauthorized access to your server, you can use our audit log to see all the records they attempted to decrypt.
Let's calculate the sha256 hash digest of your asymmetrically-encrypted symmetric DEK (symmetric_dek.asymm_enc
):
$ cat symmetric_dek.asymm_enc | base64 --decode | shasum -a 256 > symmetric_key_asymmetric_enc.digest
Because symmetric_key_asymmetric_enc.digest
is only a hash digest (and only of ciphertext whose private key is protected by your OpSep server), it's safe for OpSep to allow users to query these (without a rate-limit) to see which records may have been decrypted.
This allows you to focus your data breach response only on the minisicule percentage of users who were potentially affected!
In order to effectively use the audit log feature you will want to store this hash digest (symmetric_key_asymmetric_enc.digest
) in your database alongside your asymmetrically-encrypted symmetric DEK (symmetric_dek.asymm_enc
) with an index on this column so you can query it easily.
In the event of a suspected breach, you can query and see which hash digests your attacker attempted to decrypt.
You can manually confirm that symmetric_key_asymmetric_enc.digest
is the same as what was returned in
opsep_api_response.txt
by inspecting the two files.
The API returns asymmetric_ciphertext_sha256
as a convenience for the avoidance of ambiguity, stricly speaking your client software already has the ability to calculate this hash digest on its own.
Conclusion
While this may look complicated, client libraries make it easy for developers with no knowledge of cryptography to automagically protect their sensitive data in just a few minutes.
Here is a code snippet from our python client library:
from opsep import opsep_hybrid_encrypt_with_auditlog, opsep_hybrid_decrypt
# Testing RSA PubKey and OpSep API Server URL (normally saved in your app's config)
RSA_PUBKEY = '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA7q4R3soRD2CrjL13OK6Y\nSBG8wpjP5sbfkL0QhpJMH87grlR2SS3CUnbYCOONzQiJ3OuKAViy/lMw1KsmG9Nn\nhAot2acg1iNyZRY33LR2jwmfFF+2iRp0itPQeOHY6GS8m3WLCMtC/kWUq0Bl5g1P\nYa9JXwSkTTRJunNH0TPk8uqwFeVhpT336M1H6ed105L8a8W3mpSwlwePron7pLf7\nwD32m9RT0nNdnHBDQCsUKS/Gdp+saLYWTgj0rpnQCe8f1p3g36Gm0gTzr3X0Adow\n8gIPfxO4HU/0cdL+Pw4mpcsWJ4531taRLLGb+a2la2zAUteYcS+8d4Nb8Omkbz39\nPylvKP6R1kHElqlF3BnwUp0AdcAvOLdeX8kYUlbKE8xwjHm/KwwleKlcAZDam7hC\nRw72JUQiod0E7My+SiZ3Ij5zKnxZXmAF5BX8T+YSqSzR4Qdp2QU9L9GgAZo/HPBN\nwME9v8usjEzrEItSSg3Nn10+J+ygsCqjrCT8CnSvD8wEyDSdO/Jly9DnWJ6B2HJE\nOc4wxWGFTCE0wiQOwC3IPNxFhuWun6/4tsEQcDs5XHaBXIHry5WCiVkjwa2pc95x\niXcfoQWr1A/jLe/MrZyN4yrgDK9mmQxxNzVfLj8S9NPjJMv+K7BKvtOmvoqsf13K\n6hYJGkAdR0d99DNFlllRm7cCAwEAAQ==\n-----END PUBLIC KEY-----\n'
OPSEP_URL = 'https://test.secondguard.com/'
your_secret = b"attack at dawn!"
# Encrypt locally (symmetrically and asymmetrically) and save the results to your DB:
local_ciphertext, opsep_recovery_instructions, opsep_recovery_instructions_digest = opsep_hybrid_encrypt_with_auditlog(
to_encrypt=your_secret,
rsa_pubkey=RSA_PUBKEY,
)
# Asymmetrically decrypt opsep_recovery_instructions (via OpSep's rate-limited API) and use it to symmetrically decrypt local_ciphertext:
secret_recovered, rate_limit_dict = opsep_hybrid_decrypt(
local_ciphertext_to_decrypt=local_ciphertext,
opsep_recovery_instructions=opsep_recovery_instructions,
opsep_url=OPSEP_URL,
)
print('secret_recovered', secret_recovered)
print('opsep_recovery_instructions_digest', opsep_recovery_instructions_digest)
print('rate_limit_dict', rate_limit_dict)