Add configurable RSA-OAEP support (selectable OAEP & MGF1 digests)

[nquangit]

Summary

The RSA Encryption / Decryption operations could not reproduce RSA-OAEP where the OAEP digest and the MGF1 digest are configured independently — a common pattern in client-side encryption. For example, the widely used [node-forge] snippet below uses RSA-OAEP with SHA-256 for both the OAEP digest and MGF1:

pkey.encrypt(forge.util.encodeUtf8(plainPassword), 'RSA-OAEP', {
  md:   forge.md.sha256.create(),
  mgf1: { md: forge.md.sha256.create() }
});

Previously:

• PEM mode hardcoded Cipher.getInstance("RSA") (= RSA/ECB/PKCS1Padding) — no OAEP at all.
• KeyStore mode built the cipher from named transformations like OAEPWITHSHA-256ANDMGF1PADDING. Under SunJCE these silently leave MGF1 at SHA-1 regardless of the name, so the result didn't match forge and the MGF1 digest couldn't be controlled.

Changes

• New RsaCipherBuilder helper — single place that turns a padding (and, for OAEP, two digests) into an initialised Cipher. For OAEP it uses an explicit OAEPParameterSpec, so both the OAEP hash and the MGF1 hash are honoured. Padding and digest option lists live here, so adding a digest in
  future is a one-line change.
• RsaEncryption / RsaDecryption now expose Padding, OAEP Hash and MGF1 Hash dropdowns in both PEM and KeyStore modes. The two hash fields are shown only while an OAEP padding is selected.
• Backward compatible: padding defaults to PKCS1Padding, so existing recipes behave exactly as before; older saved recipes load unchanged.
• Resilient KeyStore padding list: sourced from the security provider but always guaranteed to include an OAEP option, because some JVMs (including Burp's bundled JRE) advertise a reduced RSA padding list that omits OAEP.

How to reproduce the forge config

RSA Encryption → Input Type PEM → paste the public key → Padding OAEPPadding, OAEP Hash SHA-256, MGF1 Hash SHA-256. (UTF-8 encoding and Base64 wrapping are left to the existing operations, e.g. To Base64.)

Testing

• mvn test — 50 tests pass.
• New RsaCipherBuilderTest covers: OAEP/SHA-256 round-trip (the forge case), OAEP randomisation, mixed OAEP/MGF1 digests, PKCS1 backward-compat, and the KeyStore padding-fallback behaviour.

[fhaag95]

Hi! Thanks for your contribution to the CSTC. We'll look into your proposed changes and will update you soon.