openssl file encryption (private encrypted RSA key and public key)

Question

I generate private encrypted RSA key:

openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out key_enc_private.pem -aes256 

Then I extract public RSA key:

openssl rsa -pubout -in key_enc_private.pem -out key_public.pem 

Using public key I encrypt single file:

openssl enc -aes-256-cbc -salt -pbkdf2 -in open.txt -out open_encrypted.txt -pass file:key_public.pem 

In order to decrypt open_encrypted.txt file, I use command:

openssl enc -d -aes-256-cbc -salt -pbkdf2 -in open_encrypted.txt -out new.txt -pass file:key_dec_private.pem 

But decryption does not work at all.

However when using following commands to encrypt and decrypt it actually works: Encrypt:

openssl pkeyutl -encrypt -pubin -inkey key_public.pem -in open.txt -out open_encrypted.txt 

Decrypt:

openssl pkeyutl -decrypt -inkey key_enc_private.pem -in open_encrypted.txt -out new.txt 

Looks like I am doing something wrong when using openssl-enc command.

Thanks for help.

Answer 1

openssl enc does not do RSA at all; it only does symmetric encryption and decryption, and normally (including your case) uses a key derived from a passphrase. What you are doing is using the first line of text in the publickey or private file as the passphrase -- and -----BEGIN PUBLIC KEY----- is different from -----BEGIN ENCRYPTED PRIVATE KEY----- so you are trying to decrypt with the wrong passphrase and it fails. (Moreover since millions of people at least know what the first lines of these files always are, your passphrases are totally insecure.)

pkeyutl does public-key cryptography, both encryption/decryption and signature/verification, including RSA; the older and now unpreferred rsautl does only RSA. But using RSA directly like this limits you to plaintexts that are smaller than the RSA modulus minus an allowance for padding, which in your case means about 240 bytes, which is typically 3 to maybe 12 lines. While some text files are that short, most are not, and then your scheme fails.

The usual way to use RSA (or other PKC) to protect non-tiny data is hybrid encryption: encrypt the data using a symmetric algorithm (which has no limit, or a very large one) and a randomly-generated single-use key, usually called a DEK (data encryption key), and encrypt that DEK (which is a fixed and small size) with RSA; the receiver then uses RSA to decrypt the DEK and uses the DEK to symmetrically decrypt the data. As noted in wikipedia, PGP does this; so does the former PKCS7 which is replaced by CMS and S/MIME; and XMLenc which is no longer much used. So do a number of other schemes various people and organizations have created, none of which has (as of now) gained very much acceptance -- except JWE, and I haven't seen it used on large data although it can be.

OpenSSL supports hybrid file encryption/decryption using either CMS or S/MIME formats, although the naming is bit confusing: openssl smime is an older function and supports both S/MIME and CMS formats, but only for the operations defined in their early versions; openssl cms is a newer function that again supports both formats but for an expanded set of operations. Since 'enveloped' -encrypt and -decrypt were in the early set, you could use either, but in general it's better to use the newer one.

Oh, and now I look at the existing Qs already suggested by Stack, this is mostly covered already by How can I encrypt a large file with OpenSSL using RSA keys? and How can I use an unsigned public key to encrypt a large file using openssl? .