Hi everyone.
I was trying to research about how to implement SSL on the traffic between my clients and the containers that I host on my server.
Basically, my plan was to use upstream SSL in HAProxy to attempt to achieve this, but in order for that to work, each individual container on my server needs to be able to decrypt SSL. I do not think that is possible and that every container has the necessary libraries for it. This puts a halt on my idea for upstream encryption of traffic from my reverse-proxy to my containers.
With that said, ChatGPT suggested I use Kubernetes with a service mesh like Istio. The idea was intriguing so I started to read about it; but before I dive head-first into using k3s (TBH it’s overkill for my setup), is there any way to implement server-side encryption with podman containers and a reverse-proxy?
After writing all of this, I think I’m missing the point about a reverse-proxy being an SSL termination endpoint, but if my question makes sense to you, please let me know your thoughts!
Thanks!
The mesh proxy would work, but it’s not easy to configure and for somewhat little benefit, especially if they’re all running on the same box. The way that’d work is, NGINX would talk to the mesh proxy which would encrypt it to the other mesh proxy for the target container, and then it would talk to the container unencrypted again. You talk to 3 containers and still end up unencrypted.
Unless you want TLS between nodes and containers, you can skip the intermediate step and have NGINX talk directly to the containers plaintext. That’s why it’s said to do TLS termination: the TLS session ends at that reverse proxy.
Thanks. As another commenter mentioned, I’m planning to deploy a reverse proxy for every pod. I’m hoping this is OK
I’m missing the point about a reverse-proxy being an SSL termination endpoint
Yes, that’s usually one of the jobs of the reverse proxy. Communication between the RP and an application container running on the same host is typically unencrypted. If you’re really paranoid about a rogue process intercepting HTTP connections between the RP and the application container, setup separate container networks for each application, and/or use unix sockets.
ChatGPT suggested I use Kubernetes
wtf…
Hey, thanks for your comment. Could you explain a bit more about how using Unix sockets would improve my security posture here (in terms of not having unencrypted traffic on the network)? I will think about creating separate namespaces in podman.
Good thing I asked haha. Is the fact that I mentioned ChatGPT setting a wrong impression? I like to go and ask about such questions to ChatGPT/Bing, sometimes they give wonderful answers, sometimes, not the best. Like this one. I thought that there must be an easier way to secure my traffic/do as much as possible to restricting it without jumping straight to k3s.
Thanks!
Nothing wrong with asking LLM’s about topics, I’d even say it’s a good idea instead of directly asking on a forum. Just like searching before asking, asking an LLM before asking humans is good.
And mentioning where you got the recommendation for k8s is also helpful. I’m not knowledgeable about k8s, but I guess the “wtf” was about the overkill of recommending k8s when simpler solutions exist.
Unix sockets have permissions like any file, so it’s simple to restrict access to a user/group and thus process running as the user. If it’s unencrypted http on a server other processes could listen on localhost, but I’m unsure about that part.
Sorry for replying this late; I wanted to read more about Unix sockets and podman before I got back. Thanks for your comment.
I already responded to the other commenter with what I’ve understood and my plans, I’ll paste it here too:
If I understand correctly, Unix sockets specifically allow two or more processes to communicate amongst each other, and are supporter on Podman (and Docker).
Now, the question is: how do I programmatically programmatically utilise sockets for containers to communicate amongst each other?
I was considering a reverse proxy per pod as someone else suggested, since every podman pod has its own network namespace. Connecting between pods should likely be through the reverse proxies then. I just need to figure out how I can automate the deployment of such proxies along with the pods.
Thanks again for your comment, and please let me know if I’m missing anything.
Thanks for the long reply. Sadly I don’t know enough about unix sockets and docker/podman networking to help you.
I’ve only used unix sockets with postgresql and signald. For both I had to mount the socket into the container and for the postgres I had to change the config to use unix sockets.
I see. My use-case would probably be better served through a software bus implementation (how would I keep all of these containers attached to the bus? Isn’t that a security risk?), but perhaps handling everything through the network behind individual reverse-proxies might be the best idea in this case.
Is the fact that I mentioned ChatGPT setting a wrong impression?
Not at all, but the fact that it suggested jumping straight to k8s for such a trivial problem is… interesting.
how using Unix sockets would improve my security posture here
Unix sockets enforce another layer of protection by requiring the user/application writing/reading to/from them to have a valid UID or be part of the correct group (traditional Linux/Unix permission system). Whereas using plain localhost HTTP networking, a rogue application could somehow listen on the loopback interface and/or exploit a race condition to bind the port and prentend to be the “real” application. Network namespaces (which container management tools use to create isolated virtual networks) mostly solve this problem. Again, basic unencrypted localhost networking is fine for a vast majority of use cases/threat models.
Hey, thanks for your comment. My apologies in replying this late; I wanted to read more about Unix sockets and podman before I got back.
If I understand correctly, Unix sockets specifically allow two or more processes to communicate amongst each other, and are supporter on Podman (and Docker).
Now, the question is: how do I programmatically programmatically utilise sockets for containers to communicate amongst each other?
I was considering a reverse proxy per pod as someone else suggested, since every podman pod has its own network namespace. Connecting between pods should likely be through the reverse proxies then. I just need to figure out how I can automate the deployment of such proxies along with the pods.
Thanks again for your comment, and please let me know if I’m missing anything.
if I’m understanding your question correct, you are trying to use tls on containers that may not have tls libraries?
there are two ways to that. one is to rebuild every container by yourself modifying its services to contain tls. the other is to use a pod. you put your service container and a reverse proxy into the same pod, setup that reverse proxy correctly as an edge proxy terminating tls, and expose only the reverse proxy’s port. that way, it will just look like a service with tls enabled.
since you are considering tls for everyone, I assume that you don’t care about overheads. adding a reverse proxy in front of every container is like 10-50MB of additional memory, and it won’t matter on modern systems.
Thank you, this is an excellent idea. I will probably not run a pod for every container (technically I can, since Netavark is supported for rootless containers in Podman 4.0), but I will definitely have a few pods on my system, where I can definitely use a reverse-proxy for every pod. Just need to figure out how I can automate it.
Thanks again
Single node k3s is possible and can do what you’re asking but has some overhead (hence your acknowledgment of overkill). One thing i think it gets right and would help here is the reverse proxy service. It’s essentially a single entity with configuration of all of your endpoints in it. It’s managed programmatically so additions or changes are not needed to he done by hand. It sounds like you need a reverse proxy to terminate the TLS then ingress objects defined to route to individual containers/pods. If you try for multiple reverse proxies you will have a bad time managing all of that overhead. I strongly recommend going for a single reverse proxy setup unless you can automate the multiple proxies setup.
Why not just run a reverse proxy container on the server hosting the rest?
That’s what I’m doing, but I would have liked encrypted communications between the other containers and the proxy too
But why?
I do not prefer to not have much, if any unencrypted traffic on my network
The traffic between containers should not be done over your normal network, you should set up a private network between containers. That way it won’t matter if it’s not encrypted.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters HTTP Hypertext Transfer Protocol, the Web HTTPS HTTP over SSL SSL Secure Sockets Layer, for transparent encryption TLS Transport Layer Security, supersedes SSL k8s Kubernetes container management package
5 acronyms in this thread; the most compressed thread commented on today has 9 acronyms.
[Thread #264 for this sub, first seen 6th Nov 2023, 14:30] [FAQ] [Full list] [Contact] [Source code]
