Monday, July 07, 2014

RSA OAEP padding with SHA512 hash algorithm

Recently I wanted to encrypt a message with RSA with OAEP padding. I also wanted to use SHA512 as hashing algorithm and mask generation function(MGF) in the OAEP padding instead of SHA1. But it looks like it is not possible with the OpenSSL/libcrypto as the SHA1 hash algorithm is hard coded in OAEP padding implementation. This is confirmed by this thread in OpenSSL forum. Though the forum thread was written around 2012 but still I couldn't find a way use either SHA256 or SHA512 as my hashing algorithm and MGF in OAEP padding.

As suggested by "Dr Stephen N. Henson"(the core developer of OpenSSL) in the forum thread , I've took the implementation of RSA OAEP padding and modified to use SHA512 instead of SHA1. It is mostly just find EVP_sha1 and replace with EVP_sha512. We also need to update the usage of SHA_DIGEST_LENGTH macro to SHA512_DIGEST_LENGTH to reflect the output length of SHA512 hash. Below is the modified RSA OAEP padding implementation which uses SHA512 algorithm. Hope it helps, cheers.



5 comments:

Moshe Wiener said...

Hello Sivachandran,
Thank you very much for publishing your example code for RSA mgf1 padding and hash 512.
Yet, I'm not clear how should this function be used. If I want to sign a message with my private key, how should I call your function? Where do I provide the RSA? etc.
I would be very grateful if you could help me with this point.
All the best,
Moshe

Sivachandran Paramasivam said...

@Moshe The second paragraph of the blog has explanation of how to incorporate the changes within OpenSSL. Basically you need modify OpenSSL source to use SHA512 and build it. The above code snippet is nothing but a modified version of OpenSSL source. Let me know if you need further help.

Moshe Wiener said...

Thanks for your reply.
My confusion is, how should I call the manipulated function (I see that RSA_padding_add_PKCS1_OAEP_SHA512 is a variation of RSA_padding_add_PKCS1_OAEP_mgf1) from my application. I fail to understand how to use the arguments of this function. Could you please tell me what is the 'param'argument for, and where do I provide the RSA key?

Sivachandran Paramasivam said...

@Moshe This function pads the data before you encrypt your data with RSA key. So you just need to pass the data(2 & 3rd params) and a buffer(1 & 2 params) to receive the padded data. I think the 'param' argument acts like a salt. In my usage I am passing NULL and 0 for param and plen respectively. Once you padded the data then you can encrypt it with RSA key. But make sure you are specifying the padding flag as 'RSA_NO_PADDING' as data is already padded.

Moshe Wiener said...

Thanks very much. I'll try it...