Overview: what configuring RSA means and when to do it
RSA configuration usually refers to generating and installing RSA key pairs for authentication or encryption: ssh public-key logins, tls/https certificates, or application keystores. The same basic steps repeat in each context , create a private key, derive a public key or CSR, install the public material where it is trusted, and lock down the private key. The examples below use openssh, OpenSSL, and common server software so you can follow along on linux and similar systems; Windows-specific notes appear where helpful.
What you need before starting
You need command-line access on the client that will hold the private key and administrative access to any server where you will install the public key or certificate. Common tools are ssh-keygen (OpenSSH), ssh-agent/ssh-add, OpenSSL, and utilities like puttygen or keytool for Windows/Java workflows. Decide a target key size (2048 bits minimum; 3072 or 4096 recommended for RSA) and whether you’ll protect the private key with a passphrase or store it in a hardware module or secrets manager.
Step 1 , Generate an RSA key pair (SSH-focused)
For SSH logins, generate a new RSA key pair with ssh-keygen. Modern OpenSSH offers options to make the private key format more secure against brute-force attempts and to use a stronger KDF. The command below produces a 4096-bit RSA private key and a matching public key. Use a descriptive comment so you can identify the key later.
ssh-keygen -t rsa -b 4096 -C "your_email@example.com" -f ~/.ssh/id_rsa -o -a 100When prompted, supply a strong passphrase if you want an extra layer of protection. The private key will be in ~/.ssh/id_rsa and the public key in ~/.ssh/id_rsa.pub.
SSH agent and adding the key
To use the key without entering the passphrase every time, add it to ssh-agent (only do this on a trusted machine):
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_rsaStep 2 , Install the public key on the server (SSH)
On the server, create the .ssh directory if needed and append your public key to authorized_keys. Correct permissions are critical , SSH rejects keys when permissions are too open.
mkdir -p ~/.ssh
chmod 700 ~/.ssh
cat /path/to/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
chown -R $USER:$USER ~/.sshIf you manage multiple users or automated accounts, place keys in the target user’s ~/.ssh/authorized_keys and confirm ownership matches that user.
Step 3 , Configure the SSH server
Edit /etc/ssh/sshd_config to enable public key authentication and set related options. Typical settings are shown below. After editing, restart or reload the SSH daemon.
# /etc/ssh/sshd_config
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
PasswordAuthentication no # optional: disable password logins for better security
PermitRootLogin prohibit-passwordRestart SSH (commands vary by distro):
sudo systemctl restart sshdStep 4 , Generate RSA private key and CSR for TLS/HTTPS (OpenSSL)
For web servers or services that require an X.509 certificate, create an RSA private key and a certificate signing request (CSR). Use OpenSSL to generate a 4096-bit private key and CSR:
openssl genpkey -algorithm RSA -out server.key -pkeyopt rsa_keygen_bits:4096
openssl req -new -key server.key -out server.csrYou will be prompted for subject fields (Common name/Organization). Submit the CSR to a certificate authority (CA) or sign it yourself for internal use. To create a self-signed certificate for testing:
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crtInstall the certificate in web servers
For nginx:
server {
listen 443 ssl;
ssl_certificate /etc/ssl/certs/server.crt;
ssl_certificate_key /etc/ssl/private/server.key;
...
}For apache:
<VirtualHost *:443>
SSLEngine on
SSLCertificateFile /etc/ssl/certs/server.crt
SSLCertificateKeyFile /etc/ssl/private/server.key
</VirtualHost>Step 5 , Convert key formats and import into keystores
Applications sometimes need keys in PKCS#12 or JKS formats. Convert a PEM private key and certificate into a PKCS#12 bundle, then import into a Java keystore if required:
openssl pkcs12 -export -out cert.p12 -inkey server.key -in server.crt -certfile chain.pem
keytool -importkeystore -destkeystore keystore.jks -srckeystore cert.p12 -srcstoretype PKCS12For putty on Windows, convert OpenSSH keys to PuTTY’s PPK format with puttygen:
puttygen id_rsa -o id_rsa.ppkStep 6 , Key management, rotation and security practices
Treat private keys as sensitive secrets: store backups in an encrypted location, use hardware-backed keys or a secrets manager when possible, and rotate keys on a regular schedule. Set strict filesystem permissions (typically 600 for private keys), avoid copying private keys to shared or unsecured systems, and revoke any associated public credentials when a private key is compromised.
Choose key sizes and algorithms carefully. RSA 2048 is a common minimum, but 3072 or 4096 increases future-proofing. For new SSH deployments consider modern alternatives such as Ed25519 for better performance and smaller keys; however, RSA remains necessary when compatibility with older systems or specific certificate formats is required.
Troubleshooting common issues
If SSH rejects your key, check permissions on the client and server (~/.ssh and authorized_keys), ensure the public key is exactly intact (no line breaks added), and verify sshd_config settings. Use verbose SSH output to diagnose connection problems:
ssh -v user@hostFor TLS problems, inspect the certificate chain and private key with OpenSSL:
openssl x509 -in server.crt -text -noout
openssl rsa -in server.key -checkAlso use s_client to test the server’s presented certificates:
openssl s_client -connect example.com:443 -showcertsSummary
Configuring RSA involves creating a secure private key, deriving or requesting the public material, installing the public key or certificate where it is needed, and protecting the private key with adequate permissions, passphrases, or hardware. For SSH you generate keys with ssh-keygen and put the public key into authorized_keys; for TLS you generate a private key and CSR with OpenSSL and install the signed certificate in your server. Always follow key management best practices: use appropriate key sizes, protect private keys, rotate regularly, and use strong passphrases or hardware-backed storage when possible.
frequently asked questions
1. What RSA key size should I use?
Use at least 2048 bits for current compatibility, but 3072 or 4096 bits provides stronger security and better future-proofing. For new ssh keys consider elliptic-curve keys like Ed25519 for better efficiency, though RSA remains common for certificate-based systems.
2. Can I use the same RSA key for SSH and TLS?
Technically you can reuse the same RSA private key across services, but it’s not recommended. Reusing keys increases impact if the key is compromised. Use separate keys for different services and keep TLS private keys within server boundaries or dedicated security modules.
3. How do I protect my RSA private key?
Protect the file with strict filesystem permissions (600), encrypt it with a strong passphrase, store it in a hardware security module or a secrets manager, and limit copies. Regularly audit access and rotate keys when personnel or system access changes.
4. My ssh key is ignored. What should I check?
Verify file permissions for ~/.ssh and authorized_keys, ensure the public key is complete and on one line, confirm sshd_config allows public-key authentication, and test with ssh -v to see which key the client is offering and why the server might reject it.
5. How do I convert an OpenSSL private key for Java?
Convert the PEM key and certificate to a PKCS#12 file with openssl pkcs12 -export, then import that PKCS#12 into a Java keystore with keytool -importkeystore. This preserves the private key and certificate in a format Java applications can use.
