Tech Specs

This tutorial will walk you through OpSep's hybrid encryption so that you understand the trustless nature of the protocol. In production, you will use client libraries that abstract away all of this complexity (see example at the bottom of this page).

Table of Contents

  1. Upgrade OpenSSL
  2. Grab an RSA Public Key
  3. Generate a Random DEK Locally
  4. Encrypt Your Secret Locally Using Your DEK
  5. Encrypt Your DEK with OpSep’s KEK
  6. Throw Away the DEK
  7. Recover Your DEK Using OpSep’s API
  8. Decrypt Your Ciphertext Locally to Recover Your Secret
  9. Rate Limiting
  10. Decryption Audit Log

Before we get started, it's important to understand the terms Data Encryption Key (DEK) and Key Encryption Key (KEK). We use DEKs to encrypt data, and a KEK to encrypt DEKs. We'll use a unique DEK for each record we encrypt, and the same KEK for each DEK. Your DEKs use symmetric encryption, because it is highly performant. Your KEK uses asymmetric encryption, so we can encrypt unlimited records using the same RSA pubkey (and locally without hitting an external server). Your KEK will be guarded by an OpSep server controlled by your security team (we recommend hardening this with an HSM).


Upgrade OpenSSL

This protocol requires openssl version 1.1 or greater. You can see what version you're running like this:

$ openssl version
OpenSSL 1.1.1g  21 Apr 2020

If you have an old version of openssl (or you're actually running LibreSSL), you need to upgrade. Here are instructions for macOS and for Ubuntu.


Grab an RSA Public Key

Grab SecondGuard's testing RSA Public Key. This is the key encryption key (KEK) that you're going to use to encrypt your data encryption keys (DEKs):

$ curl -s https://test.secondguard.com | python3 -c "import sys, json; print(json.load(sys.stdin)['rsaPubKey'].strip())" | awk '{gsub(/\\n/,"\n")}1' > rsa_kek.pub

Do not use this key to protect real data! The private key has intentionally been published. Each OpSep server generates their own key with strict access controls.


Your can see what the public key looks like by running this:

$ cat rsa_kek.pub 
-----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA7q4R3soRD2CrjL13OK6Y
SBG8wpjP5sbfkL0QhpJMH87grlR2SS3CUnbYCOONzQiJ3OuKAViy/lMw1KsmG9Nn
hAot2acg1iNyZRY33LR2jwmfFF+2iRp0itPQeOHY6GS8m3WLCMtC/kWUq0Bl5g1P
Ya9JXwSkTTRJunNH0TPk8uqwFeVhpT336M1H6ed105L8a8W3mpSwlwePron7pLf7
wD32m9RT0nNdnHBDQCsUKS/Gdp+saLYWTgj0rpnQCe8f1p3g36Gm0gTzr3X0Adow
8gIPfxO4HU/0cdL+Pw4mpcsWJ4531taRLLGb+a2la2zAUteYcS+8d4Nb8Omkbz39
PylvKP6R1kHElqlF3BnwUp0AdcAvOLdeX8kYUlbKE8xwjHm/KwwleKlcAZDam7hC
Rw72JUQiod0E7My+SiZ3Ij5zKnxZXmAF5BX8T+YSqSzR4Qdp2QU9L9GgAZo/HPBN
wME9v8usjEzrEItSSg3Nn10+J+ygsCqjrCT8CnSvD8wEyDSdO/Jly9DnWJ6B2HJE
Oc4wxWGFTCE0wiQOwC3IPNxFhuWun6/4tsEQcDs5XHaBXIHry5WCiVkjwa2pc95x
iXcfoQWr1A/jLe/MrZyN4yrgDK9mmQxxNzVfLj8S9NPjJMv+K7BKvtOmvoqsf13K
6hYJGkAdR0d99DNFlllRm7cCAwEAAQ==
-----END PUBLIC KEY-----


The same RSA public key can be re-used for every DEK, so this step only has to be performed once. Users typically save this public key locally in their configuration so they don't need to download it each time they want to encrypt a record in their database. Next, we'll use a unique DEK to symmetrically encrypt each record.


Generate a Random DEK Locally

The symmetric data encryption key (DEK) will be used to encrypt your secret locally. Our protocol is designed so that we can never recover your plaintext, even if our systems were compromised.

$ openssl rand -hex 32 > symmetric_dek.txt

Your key should look something like this:

$ cat symmetric_dek.txt
2c7776dacc5542cceecee5e7867afa08ca251c6d9d91f97856e9a01c6fcd64a6
(obviously, don't use this key!)

At 256-bits, this random symmetric key is quite long. An encrypted version of it will later be stored in your database and not something you ever have to remember.


Each record you want to secure in your database will be encrypted with its own unique symmetric DEK. This will be important in step 9 when we discuss Rate Limiting.


At the time you encrypt your data with a unique DEK, your OpSep server will not have access to this DEK. If you later use OpSep to decrypt your data, your OpSep server will be able to reconstruct this symmetric DEK with the data you supply. Recovering your plaintext requires both your ciphertext (which never leaves your server and your OpSep server never sees) as well as this symmetric DEK.


Encrypt Your Secret Locally Using Your DEK

OpSep works with any type of data (string, binary, integer, boolean, etc), but for simplicity we'll protect a string. We're choosing the secret Attack at dawn, but you can guard any secret you like:

$ echo "Attack at dawn" | openssl enc -aes-256-cbc -kfile symmetric_dek.txt -pbkdf2  > secret.symm_enc

If you get a Extra parameters given error, it likely means you're on an outdated version of openssl or an alternate program like LibreSSL that is being called instead. Please go back to the first step (Upgrade OpenSSL) for instructions on how to upgrade.


You can verify this does indeed appear encrypted (your result will be slightly different):
$ cat secret.symm_enc
Salted__??a3??=Hz???l'??f?\?

This is the data you would save to your database (along with symmetric_dek.asymm_enc in the next step) for later retrieval and decryption.


-aes-256-cbc instructs openssl to use AES in Cipher Block Chaining mode. While asymmetric encryption is slow and has size constraints, this is a very fast symmetric encryption algorithm that can handle large files sizes.


You can encrypt a whole file (to_encrypt.txt) by first saving it and then passing it into openssl (no need to pipe it in):

$ openssl enc -aes-256-cbc -kfile symmetric_dek.txt -in to_encrypt.txt -pbkdf2 > secret.symm_enc


Instead of -kfile symmetric_dek.txt. you can enter -k 2c7776dacc5542cceecee5e7867afa08ca251c6d9d91f97856e9a01c6fcd64a6 if you prefer. Note that the -k is lowercase, and replace the hexademical string after it with your symmetric_dek.txt generated above.


The -pbkdf2 flag is only used to surpress a *** WARNING : deprecated key derivation used error that would confuse this exercise. Without this flag, we get a warning that for (regular) symmetric encryption you want to use a key derivation function. Since our key is already a massive 256 bits long, there's no reason to suffer the performance hit of a KDF. However, for this demo we let openssl perform this unneccsary KDF for simplicity. You're welcome to get rid of -pbkdf2 throughout these examples if you like; you may then want to add 2> /dev/null to your openssl commands to silence this the error.


Encrypt Your DEK with OpSep's KEK

We're using Optimal Asymmetric Encryption Padding with RSA and SHA-256 as our hash function. Due to the magic of asymmetric cryptography, this takes place locally using the public key (rsa_kek.pub) that we downloaded earlier.

To start, we'll convert our DEK (symmetric_dek.txt) to json:

$ echo "{\"key\": \"` cat symmetric_dek.txt `\"}" > symmetric_dek.json

By using a JSON file, OpSep supports powerful restrictions on future decryption. For example, we can add an RFC3339 timestamp called deprecate_at that will mark the key as deprecated at a certain time in the future. This notation occurs client-side and does not immediately leave your client. However, if your OpSep server decrypts this file in the future and the deprecate_at date has passed, then OpSep can refuse to return this deprecated key.


Your OpSep server also supports the client_record_id and risk_multiplier (see OpSep server for details).


Now we asymmetrically encrypt symmetric_dek.json with SecondGuard's public key (rsa_kek.pub):

$ openssl pkeyutl -in symmetric_dek.json -encrypt -pubin -inkey rsa_kek.pub -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 -pkeyopt rsa_mgf1_md:sha256 | base64 > symmetric_dek.asymm_enc

If you get a parameter setting error, it likely means you're on an outdated version of openssl or an alternate program like LibreSSL that is being called instead. Please go back to the first step (Upgrade OpenSSL) for instructions on how to upgrade.


You can verify this does indeed appear encrypted:

$ cat symmetric_dek.asymm_enc

This is the data you would save to your database (along with secret.symm_enc).


Base64 ciphertext is easier to work with than binary data. Encryption performance is not meaningfully affected as this is only the symmetric DEK, which are a small fixed size (unlike your plaintext which could be very large).


While each record you want to secure will require its own unique symmetric DEK, you can encrypt all of them with the same RSA public key. The corresponding private key (rsa_kek.priv) is stored in OpSep's HSM, and useage is controlled by a rate-limit policy, access controls, and an intrustion alarm system.


Notice that all the encryption took place locally and is non-interactive, which is great for performance. If you needed to revoke decryption access during a suspected breach (or OpSep were to ever experience downtime) your regular data encryption would continue unaffected.


Throw Away the DEK

If an attacker gets your ciphertext (secret.symm_enc) and your symmetric DEK (symmetric_dek.txt & symmetric_dek.json) they can easily decrypt your ciphertext into plaintext. By throwing away your symmetric DEK (symmetric_dek.txt & symmetric_dek.json), you eliminate this risk and can truly secure your database. We'll show how to recreate it from your asymmetrically-encrypted symmetric DEK (symmetric_dek.asymm_enc) in the next step.

For the purpose of this demo we're not bothering to throw away your DEK, because it will make it slightly easier to keep track of what's happening.

If you want to throw away the DEK, enter the following:

$ rm symmetric_dek.txt symmetric_dek.json


Recover Your DEK Using OpSep's API

Let's say some time has passed and you want to decrypt your ciphertext. You make the following query to OpSep's servers to recover your symmetric data encryption key (DEK):

$ curl -s -X POST https://test.secondguard.com/api/v1/decrypt -H 'Content-Type: application/json' -d '{"key_retrieval_ciphertext":"'$(cat symmetric_dek.asymm_enc)'"}' > opsep_api_response.txt

We're using '$(cat ...)' as a fancy of way passing in the contents of symmetric_dek.asymm_enc into the json payload that we're POSTing. Normally, your client library would handle all this complexity for you.


If you prefer, you can manually substitute so that your -d flag looks something like this:

-d '{"key_retrieval_ciphertext":"jx70uVbQaZeQ...CfTIFI8iqdYX73E"}'


We can parse the symmetric DEK field from that API response:

$ cat opsep_api_response.txt | python3 -c "import sys, json; print(json.load(sys.stdin)['keyRecovered'].strip())" > symmetric_dek_recovered.txt

We can confirm this returned key matches the original by running the following:

$ diff symmetric_dek.txt symmetric_dek_recovered.txt
$
(if there were any difference between the files you'd see that displayed here)


Using python3 is a verbose way to parse the JSON that OpSep's API returns. If you have jq installed you can instead do:

$ cat opsep_api_response.txt | jq -r '.keyRecovered' > symmetric_dek_recovered.txt 


Decrypt Your Ciphertext Locally to Recover Your Secret

Now all that's left to do is locally recover your original secret with your symmetric DEK:

$ openssl enc -d -aes-256-cbc -in secret.symm_enc -kfile symmetric_dek_recovered.txt -pbkdf2 
Attack at dawn

While your OpSep server did gain access to your symmetric DEK (symmetric_dek_recovered.txt), it never saw your ciphertext (secret.symm_enc) and it has no ability to access your plaintext. In fact, it doesn't even know what type of data you're protecting (binary, string, integer, boolean, etc), which database you use to store your ciphertext, or even where it is hosted.


Rate Limiting

You may have noticed that the curl command to OpSep's servers returned the Rate Limiting fields ratelimitTotal, ratelimitRemaining, and ratelimitResetsIn. Calls to your OpSep server to decrypt your symmetric DEKs are intentionally rate limited. This means that even if an attacker or rogue employee has root access to your web server and database, OpSep's rate-limit policy allows you to configure the amount of data they'll be able to decrypt before you are notified and they are blocked.

Most of our users set their rate-limit so that their maximum exposure from a breach is < 0.1% of their data. This protection remains even if your attacker has physical control of your server! OpSep also keeps a log of every record your attackers attempt to decrypt. These rules are easy to set and adjust using our web UI with strong multi-factor authentication. We automatically alert you to suspicious decryption requests 24/7 so that you can quickly discover if your servers have been compromised.


You can see these rate limits by running:

$ cat opsep_api_response.txt

Note that the api response will be easier to read if you pipe the output to a json parser like this:

$ cat opsep_api_response.txt | python -m json.tool

Or like this if you have jq installed:

$ cat opsep_api_response.txt | jq


Decryption Audit Log

If a rogue employee or outside attacker gains unauthorized access to your server, you can use our audit log to see all the records they attempted to decrypt. Let's calculate the sha256 hash digest of your asymmetrically-encrypted symmetric DEK (symmetric_dek.asymm_enc):

$ cat symmetric_dek.asymm_enc | base64 --decode | shasum -a 256 > symmetric_key_asymmetric_enc.digest

Because symmetric_key_asymmetric_enc.digest is only a hash digest (and only of ciphertext whose private key is protected by your OpSep server), it's safe for OpSep to allow users to query these (without a rate-limit) to see which records may have been decrypted. This allows you to focus your data breach response only on the minisicule percentage of users who were potentially affected!

In order to effectively use the audit log feature you will want to store this hash digest (symmetric_key_asymmetric_enc.digest) in your database alongside your asymmetrically-encrypted symmetric DEK (symmetric_dek.asymm_enc) with an index on this column so you can query it easily. In the event of a suspected breach, you can query and see which hash digests your attacker attempted to decrypt.


You can manually confirm that symmetric_key_asymmetric_enc.digest is the same as what was returned in opsep_api_response.txt by inspecting the two files. The API returns asymmetric_ciphertext_sha256 as a convenience for the avoidance of ambiguity, stricly speaking your client software already has the ability to calculate this hash digest on its own.



Conclusion

While this may look complicated, client libraries make it easy for developers with no knowledge of cryptography to automagically protect their sensitive data in just a few minutes.

Here is a code snippet from our python client library:

from opsep import opsep_hybrid_encrypt_with_auditlog, opsep_hybrid_decrypt

# Testing RSA PubKey and OpSep API Server URL (normally saved in your app's config)
RSA_PUBKEY = '-----BEGIN PUBLIC KEY-----\nMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA7q4R3soRD2CrjL13OK6Y\nSBG8wpjP5sbfkL0QhpJMH87grlR2SS3CUnbYCOONzQiJ3OuKAViy/lMw1KsmG9Nn\nhAot2acg1iNyZRY33LR2jwmfFF+2iRp0itPQeOHY6GS8m3WLCMtC/kWUq0Bl5g1P\nYa9JXwSkTTRJunNH0TPk8uqwFeVhpT336M1H6ed105L8a8W3mpSwlwePron7pLf7\nwD32m9RT0nNdnHBDQCsUKS/Gdp+saLYWTgj0rpnQCe8f1p3g36Gm0gTzr3X0Adow\n8gIPfxO4HU/0cdL+Pw4mpcsWJ4531taRLLGb+a2la2zAUteYcS+8d4Nb8Omkbz39\nPylvKP6R1kHElqlF3BnwUp0AdcAvOLdeX8kYUlbKE8xwjHm/KwwleKlcAZDam7hC\nRw72JUQiod0E7My+SiZ3Ij5zKnxZXmAF5BX8T+YSqSzR4Qdp2QU9L9GgAZo/HPBN\nwME9v8usjEzrEItSSg3Nn10+J+ygsCqjrCT8CnSvD8wEyDSdO/Jly9DnWJ6B2HJE\nOc4wxWGFTCE0wiQOwC3IPNxFhuWun6/4tsEQcDs5XHaBXIHry5WCiVkjwa2pc95x\niXcfoQWr1A/jLe/MrZyN4yrgDK9mmQxxNzVfLj8S9NPjJMv+K7BKvtOmvoqsf13K\n6hYJGkAdR0d99DNFlllRm7cCAwEAAQ==\n-----END PUBLIC KEY-----\n'
OPSEP_URL = 'https://test.secondguard.com/'

your_secret = b"attack at dawn!"

# Encrypt locally (symmetrically and asymmetrically) and save the results to your DB:
local_ciphertext, opsep_recovery_instructions, opsep_recovery_instructions_digest = opsep_hybrid_encrypt_with_auditlog(
    to_encrypt=your_secret,
    rsa_pubkey=RSA_PUBKEY, 
)

# Asymmetrically decrypt opsep_recovery_instructions (via OpSep's rate-limited API) and use it to symmetrically decrypt local_ciphertext: 
secret_recovered, rate_limit_dict = opsep_hybrid_decrypt( 
    local_ciphertext_to_decrypt=local_ciphertext, 
    opsep_recovery_instructions=opsep_recovery_instructions,
    opsep_url=OPSEP_URL,
)

print('secret_recovered', secret_recovered)
print('opsep_recovery_instructions_digest', opsep_recovery_instructions_digest)
print('rate_limit_dict', rate_limit_dict)