If you want to get fancy: systemd credentials. It can store the secrets encrypted on disk and seal the encryption key with the TPM chip. The encrypted secret is decrypted (non-interactively) and made available only to a specific systemd service. The process itself does not special systemd integration–it just sees a plain text file containing the secret, backed by a tmpfs that’s not visible to other processes.

Depending on which TPM PCRs you bind to, you can choose how secure you want it to be. A reasonable/usable configuration would be something like binding to PCRs 7 and 14. With that setup, the TPM will not unseal the key if the system is booted into any other OS (i.e. anything signed with a different UEFI Secure Boot key). But if you really want to lock things down, you can bind to additional PCRs and make it so changing any hardware, boot order, BIOS setting, etc. will prevent the TPM from unsealing the key.

Yep, this is a root of trust problem. Your choice will ultimately come down to how much you want to invest and how much inconvenience you’ll put up with, measured against how secure you want it to be.

Personally, I go for full disk encryption and then just store things on the filesystem in secure (to the OS) ways. File permissions and users and groups, etc. Most other things boil down to that though something like vault adds a layer of access control in that you can seal it off in the case of a breach (if you care) and can get granular with authz permissions in a centralized place, only managing authn in your distributed tools.

There’s probably some ideal system out there like vault but with a plugin that can ping your phone for quick verifications that would likely be ultra ideal, but I haven’t seen that. Personally I’d love something like that.

Environment variables. If they’re in my network, that has no open ports to the internet, I’ve got plenty more problems.

Even a dev machine, think about how many env vars a normal dev has, plenty to loose.

Secrets management tool for self hosting on my level would bring more complexity for little gain.

Bash scripts etc can be uploaded to open repo and not share secrets which is what I want.

Ideally, a secrets manager that you can unlock once and then give access to the secrets. You can either unlock it at boot by entering the password, or if you have a TPM, you can also do something to encrypt the main key with the TPM and then when you boot up, if all secure boot checks passes, you can decrypt that key with the TPM and unlock the secrets manager in question.

You can also store the secrets on another machine that only exposes say, the Vault API. Like a dedicated Raspberry Pi just for this function where remote access is disabled and everything, it only serves secrets. That way, you can trust the Vault logs to know when each secret was accessed and find anomalies this way.

If someone breaks in via SSH or whatever, your security already fell apart. That box can no longer be trusted for anything. Especially if they breached the root account. Doesn’t matter how big the fortress is, if you’re inside, it’s game over, it’s time to evaluate the damages and clean up.

Hashicorp Vault + Vault Config Operator + external-secrets. I have a simple chart that can add credentials to different apps which mostly gets used in argocd with its multichart functionality. A simple bash script to create the vault policies, which use the k8s back end to allow auth.

I use shh keys for all my remote machines, set passwords automatically with ansible, and store them with pass.

https://www.passwordstore.org/

EDIT:

Just to clarify, ansible can use pass as a password store, so in the ansible playbooks you can write which password you want to retrieve from pass.

You can also call pass from any shell script by writing $(pass <target_password>)

I generate a unique key pair (or token) for each service that I want to access from the host machine. I see no issue with storing that dedicated private key locally in plaintext (obviously in a folder where only the required user can read it and I except it from backup and versioning). I use one dedicated user per externally accessible service.

Should the machine itself become compromised this would indicate that my personal master key and master password have been compromised or someone gained physical access. That would require me to restart from a blank page anyways.

Yeah, I haven’t gotten past using plaintext secrets in separate files (eg. using Home Assistant’s secrets files). I do the same for any Python scripts that need secrets too, like Slack auth tokens and stuff.

I haven’t really gotten around to looking into secrets management in my homelab, and I know I really should. As much as I have a lot of faith in my Nginx and Authelia config, it only takes one hole for someone to get in and get to those secrets files, especially if that hole is a security flaw in something like Home Assistant - one of the very few services I can access publicly, without using my Wireguard VPN.

Not ideal but a few scattered ash agents I unlock on reboot. And zabbix to tell me if they’re not running.

I haven’t played around with it much, so this is a pretty uninformed suggestion, but BitWarden have a secrets manager, currently in beta. You can create projects, then grant human or service accounts access to secrets. Looks promising, but I have to say I’m a bit of a BW fanboy so I concede some bias.

I’m on Kubernetes with ArgoCD gitops.

I use sealedsecrets, so all my secrets are in git encrypted. The encryption key is in my keepass vault.

• For kuberentes stuff, I use external-secrets-operator. It can be paired with selfhosted stuff, like Hashicorp Vault.
• For machines, disk encryption LUKS, and then files in plain text, with minimal permissions: mode 400.
• For my laptop, keepass.

In my opinion, for home selfhosted stuff you don’t have to go for complex solutions. In the industry, the problem is that secrets needs to be served to different systems, by different people, with some kind of audit logs. Unless you are working with lots of people, environment variables are OK. You github/gitlab may have all scripts with variables, and your disk may have a .env file with mode 400. If you make any machine or container with a single responsibility, there should be no secret leaks among them.

For example, let say your wordpress instance gets pwned. It should only have its needed secrets (like its db credentials), so your wikimedia instance is still fine.

Two more options you might consider:

• secret-tool - like a vault that unlocks when a user logs in to their session. This shifts the problem to keeping the user’s login credentials secure but depending on your setup that might be preferable. Just be aware the once unlocked any process could access the vault in theory (I wish they’d add access controls…)
• podman secrets - so you can securely provide secrets to containers. You can set these once securely then nothing except processes in the container can get them.

Generally you want to model out what the risks are and how acceptable each risk is, compared to the effort to mitigate it.

Physical access, where someone can break in and get access to your secret material. So, you can just have full disk encryption to mitigate that. If you want to remotely reboot, you can install dropbear with SSH keys.

If you want to control and secure remote access you care about authentication and authorization, as well as the data in transit. To protect the data in transit you need to ensure it’s encrypted, so use SSH (or a similar encrypted protocol).

For auth and auth you want to minimize the risk. You can control who can log-in and from where with allow-lists of IP ranges, or control their keys. For example, for external SSH access I only use keys that are physically backed, either through a Yubikey or secure-enclave generated key. Those are non-extractable, so the only machine it can come from is the ones holding it. No one can get in without stealing the device. The device itself is full-disk encrypted, or course. You can also control how the keys are used, either biometric auth, password auth, or both, if you want.

Then, within your network for automated job you can setup ssh certs between accounts that have no other access other than to run the specific task. If you need further abilities, run those locally based on a trigger of the transfer itself. Sure, the cert used is in memory, and on disk, but it’s encrypted on the disk. To be read from memory you’d need a very skilled attacker.

At that point you’re worried about someone physically breaking it and reading memory in a running system. That’s a deeply unlikely scenario. The other risk is someone breaking SSH cert-only auth, with strong certs, and that’s only if you have SSH publicly open.

Turn off external access to everything and you’re back to local network attacks, which you can mitigate with: password controlled wifi, MAC control if you want (although can be spoofed), and even then you can limit communication on your local network. You could vlan out the jobs you care about to their own network which isn’t wifi accessible.

Basically it comes down to making the cert/password low-risk if you lose it, and then mitigating everything to that point. Then not worrying.

This is something I’ve always pondered on myself when it comes to automation as it really does seem that once someone gains access, it really is just a matter of time for the to figure out where credentials are stored.

However, this is possibly already a thing or it’s not because maybe it doesn’t sound like a good idea?? But I was just reminded of how we use MFA, and it dawned on me how these ideas behind automation and how to pull credentials we are always thinking about how we get the credential for the script so it can use it, but what if we had a way for the automation to prompt and ask us for the credential? Because surely your automation should be running on a regular schedule, where you expect these prompts. In turn the idea of an MFA prompt could also be automated?

Not sure if that’s a good idea or I’m just over complicating things.