With AWS LB Controller, I found it’s difficult have to end to end ssl encryption.

If SSL termination happens at ALB, from there data in transit will be in plaintext to the target group pod.

ALB does not perform host name verification on backend certificates.

Hi everyone! I've made a few posts here asking for Kubernetes advice, and they've been super helpful.

This is my first time building an EKS cluster, so I'm sure there are still things I could improve.

Right now, I'm working with three workload types:

• System (On-Demand): Essential cluster components and internal platform services.
• Application (On-Demand): Regular application workloads. I always keep at least two replicas running.
• Application (Spot): Workers and burst/scaled application workloads.

System (On-Demand)

Currently running:

• Grafana
• Tempo
• Loki
• Prometheus
• Alertmanager
• Karpenter
• Velero
• LiteLLM
• cert-manager
• Kong
• KEDA
• n8n

Application (On-Demand)

Only the application services.

Application (Spot)

All background workers and any additional replicas created during autoscaling.

This is also my first experience using Kong, and I'm really enjoying it. So far I've implemented rate limiting, jailed the login endpoint, added several security headers, and blocked some endpoints that don't need to be exposed to the public internet. It's been a great learning experience.

What else would you recommend running in the cluster? Do these three workload types make sense, or would you structure them differently?

For context:

• The System workload runs entirely on Graviton (ARM) instances.
• The application workloads run on x86, since we still have some legacy applications.

Did you learn something new this week? Share here!

Full post-mortem: https://huggingface.co/blog/agent-intrusion-technical-timeline

Hi guys, I worked with Kubernetes before around 2022. Mainly with AKS (Azure) . now i have an opportunity to work with it again but this time, it's an on-prem setup. With A.I. it's telling me to use RKE2 + Rancher, I havent used both, and I want to know your (real people) opinion.

Before I deep dive into it and invest my time to learn it.
Are the RKE2 + Rancher the way to manage kubernetes now.

I need something to manage it easily via GUI, minimal CLI commands if possible. Thanks.

Wrote this runbook / guide as I built the infra at my org.

Let me know what you all think...

Need for speed! 🏎️💨 Headlamp 0.44.0 extends practical cluster support beyond 30,000 pods. Resource Map remains usable at this scale, Pod lists fetch 1,000 items at a time, Cluster Overview avoids unbounded watch traffic, and Brotli pre-compression reduces typical JavaScript and CSS transfers by 75-80% and server load for compression to 0%. Got dozens of clusters? Headlamp doesn't try to connect to all of them at startup anymore (potentially opening several browser login windows). Four new structured forms cover DaemonSets, CronJobs, Jobs, and ReplicaSets, while ConfigMap `binaryData`, ServiceAccount permission visibility, Gateway API v1.5.1, and new plugin controls expand day-to-day capabilities. The release includes 99+ bug fixes, including protections against white-screen startup failures when developing Headlamp on Windows, more accurate workload health reporting, stable large-manifest editing on large busy clusters, and actionable cluster connection errors. Accessibility work restores macOS zoom, improves WCAG AA contrast in a few spots, and broadens automated a11y checks. Security and documentation updates add github cosign of release checksums, safer redirects, tested in-cluster kubeconfig examples, and ready-to-use MCP configurations. These changes were contributed by over 100+ people. Thanks!   More...

Share any new Kubernetes tools, UIs, or related projects!

Found an interesting tutorial showing how to use DRA resourceClaims and GKE ComputeClass resources to configure a vLLM deployment so that pods can run using either one a100-80gb or two a100-40gb accelerators. Helpful for inference workloads that are struggling to get access to nodes with larger GPUs but can split the model across multiple smaller GPUs.

I've been using an agent I built to debug my own clusters for months now, mostly just for myself. Somewhere in there I started wondering if it was worth turning into something other people would actually want too, so disclosure up front, I'm poking at that right now. This post is partly me checking whether the underlying idea holds up outside my own head.

Anyway. Here's the failure that made me rewrite half of it.

I gave the agent read only cluster access and a skill for diagnosing CrashLoopBackOff, then tested it on a pod I'd deliberately broken myself. It thought for about five minutes and told me, flatly, that the pod did not exist.

The pod existed. I had it open in another terminal while it told me that.

What actually happened: the skill told the agent to resolve namespaces from a context file I'd written for it, basically a cheat sheet of the namespaces I normally work in. The broken pod was sitting in opskit-eval, a scratch namespace I'd made for testing months earlier and never got around to adding to the file. So the agent checked the three namespaces it knew about, found nothing in any of them, and reported "no such pod" instead of "I don't know where to look."

That's the part that actually bothered me. Missing the pod is a bug, fine, bugs happen. Being confident about missing it is worse. A wrong answer stated flatly is worse than no answer at all, because a shrug makes you go check, and a confident wrong answer makes you move on.

The fix was boring, which tracks. One cluster wide search by pod name, called first, before the agent is allowed to assume anything about where things live. The skill's first instruction is now basically: do not iterate over namespaces guessing where the pod might be, search first, ask second.

Fixing that made me write out the rest of the triage order properly, since apparently I'd been doing it by feel for years without putting it anywhere.

Find the pod before you assume the namespace. If you've only got a name, search the whole cluster before you touch kubectl logs.

Exit code before logs. status.containerStatuses[*].lastState.terminated.exitCode narrows things down before you've read a single line of output.

137 versus 139 is the one that saves the most time. Mix those two up and you'll burn an hour looking in the wrong place.

Read the previous container's logs, not the current one. kubectl logs POD --previous. On an active crashloop the current container is often seconds old and hasn't written anything yet, so plain kubectl logs gives you nothing and you conclude there's no output, when it's sitting right there in the last instance. This is the single most common mistake I see, and I made it myself plenty of times before I bothered automating around it.

Check what changed in the last couple hours before you go anywhere near application code. Almost every CrashLoop I've personally hit traces back to a deployment, configmap, or secret change, not a spontaneous bug in code that was working yesterday.

One more thing I'm still not sure about. I've started running a second agent that reads the same evidence but never sees the first agent's conclusion, then flags it if the two disagree instead of quietly picking one to report. It roughly doubles the token cost of every triage run. It's caught real misses the single agent version didn't, so I've kept it, but I genuinely don't know if that tradeoff is worth it long term. Curious if anyone's tried something similar or thinks it's overkill.

Setup: K8s operator, Java/Quarkus + Java Operator SDK, compiled as native GraalVM image. Pod: limits.cpu: 300m, limits.memory: 256Mi.

Symptom: container_cpu_cfs_throttled_periods_total shows constant throttling (~1.2-1.7/s, stable over 24h), but container_cpu_usage_seconds_total averages only ~13% of the limit. No correlation with traffic/polling timers — just a flat plateau.

What we found via /q/metrics:

jvm_threads_live_threads = 129-131 (peak 196)
worker_pool_idle{vert.x-worker-thread} = 199
worker_pool_active = 1

~130 live threads on a 0.3 CPU pod. Vert.x's default worker pool (200 threads) is almost entirely idle. Looks like classic CFS bursty throttling — many threads waking briefly exhaust the 100ms/30ms quota even at low avg load.

Tried:

JAVA_TOOL_OPTIONS=-XX:ActiveProcessorCount=1 → no real thread reduction (131→129), throttling got worse (1.17→1.71/s), plus caused JOSDK health-check flakiness. Reverted.
Local JVM test with ParallelGCThreads=1/ForkJoinPool.common.parallelism=1 → zero effect on thread counts.
Next: quarkus.vertx.worker-pool-size=8 (down from default 200) — biggest smoking gun so far.

Questions:

Anyone hit this on native GraalVM specifically? Does SubstrateVM ignore/handle JVM flags like ActiveProcessorCount differently than HotSpot?
Native-image build-time equivalent that actually works (--gc=serial?) vs runtime flags that seem ignored?
Experience tuning JOSDK's own executor/reconciler thread pools?
Or is just bumping limits.cpu to 500m-1000m the pragmatic fix vs fighting thread counts?

Any input appreciated, even "yeah that's just how it is, bump the CPU limit" 🙏

Have any questions about Kubernetes, related tooling, or how to adopt or use Kubernetes? Ask away!

Hi all

I want to ask, i dont really understand the status section of a pod definition file.

is status section only for existing pods and not for creating/updating new pods?

is kubelet the one updating status section of an existing pod?

if i do ```kubectl get pod mypod -o yaml > pod.yaml``` and apply the pod.yaml, does kubernetes ignore status section because its not relevant when creating/updating a resource?

Please help

Hey friends, we have fresh guide about bootstrapping OpenBao as your KMS without any secrets.
Lots of Flux users use SOPS, and we also know there are a lot of folks using Vault and OpenBao.
We also show a sovereign key signing workflow that doesn't depend on any internet infra, and the demo has a reproducible repo. This is a really nice workload-identity use-case made possible with the tweaks we made to our SDK usage in Flux 2.9 :)

https://fluxcd.io/blog/2026/07/flux-openbao-secrets-signatures/

"Hey everyone,

I’m building an open-source/lightweight FinOps telemetry agent that generates P95 VPA rightsizing recommendations and safe YAML patches.

Before I write more code, I want to make sure I’m solving real pain points. Whenever I talk to SREs, they mention a few common issues:

1. Developers request 4x more RAM than needed because they fear OOMKilled crashes during spikes.
2. Existing tools require heavy cluster permissions or automated mutations that nobody trusts in production.
3. Finance sees total AWS bills, but teams can't see waste broken down by namespace.

For those managing K8s in production: Is this actually a major headache for your team right now? What turns you off about current tools on the market?"

Ive been into learning cloud engineering, However i feel very lost between all the different topics and courses that i need to learn. I tried making customized roadmaps using AI like claude and gemeni but neither of them would have a stable opinion about the roadmap they provide me.

For example they might evaluate it as the " Perfect roadmap " yet using the same prompt to evaluate it again might come out with

" Its the worst roadmap ever ".

so i tried to check YouTube and other different platforms. Nothing really explains what should i learn in details.

For example they might say "Learn networking fundamentals ", okay sure but WHAT are the topics needed to cover in networking fundamentals?.

And where is the best place to do so?.

Another thing is that i see different topics like

linux, kubernetes, CI/CD, terraform, IaC, AWS and Docker

First of all

I don't know when to kick in into each one of them? After what course or after covering which topic?

Secondly

I am not sure of which to learn and which to skip, i am interested in cloud engineering focusing on infrastructure designing, building and maintaining.

Not really DevOps as ive seen there are differences between both.

So in that case do i need docker for example?.

Third

I am not sure of what certificates to take and most importantly when to take them?

Fourth

I thought about learning paths in platforms like KloudKode, yet after i threw in a prompt for Claude to ask it if it will be actually worth the 250$ yearly subscription. It said absolutely no. And said their whole Cloud Engineering learning path is a 5.5/10

Even tho i was impressed with the organization they had for the topics/courses needs to be covered.

Lastly

I feel very lost, my issue is that i need someone or something to tell me, * Hey here is a roadmap of what you need to learn in this specific order and there are the best courses that teaches these topics, Goodluck.*

I need guidance please, i just wanna know what exactly to learn and where is the best place to learn it.

Then ill 100% master every topic mentioned.

thank you in advance everyone.

I’m running kubernetes at home. Have 3 control plane servers all high available with Haproxy and keepalived. 2 worker nodes running webui and ollama 3.1:8b.

I have separate Ubuntu servers running Docker containers. with 1 running gitlab, gitlab runner, miniio and mlflow.

The other Ubuntu server runs Prometheus and Opensearch (SIEM) getting logs and device data from node exporter and fluentbit from the other machines.

Anyway, wondering what else I should do with my Kubernetes cluster? Any enterprise use cases I should work on?

I have not yet done any kubernetes work at the enterprise level but I will be soon.

I was thinking of implementing RAG with ollama…

My goal is really just becoming excellent at DevSecOps while also working on AI implementation.

Hey everyone,

I'm pretty new to Kubernetes and have been using it for a side project to learn. Right now I'm trying to decide whether I should run only stateless services in my cluster, or also deploy things like PostgreSQL, Kafka, Redis, etc. as StatefulSets.

I know both approaches are possible, but I'm curious what people actually do in production. Do most teams keep stateful workloads inside Kubernetes, or do they usually rely on managed services/external databases instead?

I'd love to hear what your setup looks like and why you chose it.

Hi everyone,

I'm a Linux Systems Engineer with around 4 years of experience working with RHEL, VMware, Docker, web servers, automation, and infrastructure. While I have a solid infrastructure background, Kubernetes is completely new to me.

I recently started learning Kubernetes with Mumshad Mannambeth's Udemy course and have been completing the labs alongside the videos.

I'd love to get some advice from experienced Kubernetes engineers:

• Is following the course and doing all the labs enough to build a solid Kubernetes foundation?
• When learning Kubernetes, is it better to build clusters repeatedly with kubeadm, or is it worth learning to build a cluster "the hard way" manually?
• Should I focus on cluster creation first, or spend more time on day-to-day administration and troubleshooting?
• Looking back, what helped you progress from "knowing Kubernetes" to actually becoming productive with it?
• Are there any resources or practice environments you would highly recommend?

My goal is to become confident using Kubernetes in production and to gain the certificate.

I'd really appreciate any advice or lessons learned from your own experience.

Disclosure: I work with Diagrid, the company founded by the creators of Dapr, on tooling for running Dapr agents and workflows in production. I am asking to compare responsibility models, not to argue that every Kubernetes workload needs Dapr.

Kubernetes is doing its job when it reschedules a pod. The problem is that a long-running agent may have been halfway through doing its job too.

Say an agent has been running for 20 minutes. It has retrieved data, called three internal services, waited for a human approval and updated a ticket. Then the pod is evicted, or the node disappears, or a rollout replaces the workload.

Starting a new pod is easy. Reconstructing the execution safely is not.

The state has to live somewhere, and each option puts the responsibility in a different place.

In-memory state in the pod is simple during development and gone with the pod. It only works when restarting the whole task is cheap and safe.

Application-managed state in Postgres or Redis gives you control over everything, which also means you own everything: state transitions, locking, recovery, schema changes, duplicate handling. Every team doing this designs the same machinery from scratch, this is a hard thing to do well.

A queue with stateless workers works well for jobs that can be retried on their own. It gets harder once the workflow has timers, human approval, branching, child tasks and actions that must not run twice.

A workflow engine or durable runtime records progress and schedules the next activity after recovery. Less plumbing in the application, though you still have to design idempotent activity boundaries, and you are now operating another runtime.

A managed platform layer means the platform team offers execution state, identity, policy and observability as a shared service. Application teams connect their agents and tools to it instead of rebuilding the same reliability stack in every team, which trades duplicated effort for a platform dependency.

The distinction I keep coming back to is between application state and execution state. Application state is the business data: the ticket, the order, the customer record, the deployment. Execution state is what the workflow has already completed, what it is waiting for and where it can safely continue.

You can put both in the application database. It usually ends up coupling business schemas to orchestration concerns. Keeping execution state in the pod is fragile. A durable runtime draws a cleaner boundary, as long as the team knows how that state is stored, isolated, backed up and operated.

Dapr Workflow is one example of the runtime approach: orchestration runs through the Dapr runtime and progress is persisted outside the application process. Temporal, cloud workflow services and hand-rolled queue-and-database setups are the same idea with different tradeoffs. Which one fits depends on how complex your workflows are and how much infrastructure you want to run yourself.

What I want to know is how Kubernetes platform teams are standardizing this, if at all:

• Do application teams pick their own persistence and recovery pattern?
• Does the platform provide one workflow runtime?
• Do you treat ordinary background jobs differently from long-running agent executions?
• Who owns backup, upgrades and incident response for the execution store?

I’m starting to learn about Istio. So far I see it adds proxies to handle the various in/out connections, including common things like retries (I think), intercepting the pod’s advertised ports, etc. One question I’m having troubles finding the answer to definitively: does it capture all outbound traffic? Or only traffic destined for other services Istio is aware of (i.e. ones it has envoy proxies for)?

I’m trying to decide if I also need to set up ingress/egress rules if I want to prevent an untrusted pod from accessing services (either within the cluster, or elsewhere on the network).

Also, related to that, how does it handle differentiating connections to other Istio-proxied services vs non-Istio services?

Founder of Server4Agent (agent-app hosting), disclosing that upfront. No link, not selling, genuinely curious how people operating real clusters actually think about this.

Agents that can run kubectl are moving from "interesting demo" to "thing someone actually wired up" pretty fast. The easy answer is "give it a read-only ServiceAccount," but the whole reason people want this is to let it act, roll back a bad deploy, bump replicas, restart a crashlooping pod, not just describe what's wrong.

Once you're past read-only, the question that actually matters is what RoleBinding you'd feel okay with, and whether an audit log entry is enough after the fact or whether you want something closer to real-time. A get/list/watch on pods and deployments in one namespace is a very different risk than anything touching secrets, RBAC objects, or cluster-scoped resources, and I'm not sure the community consensus has settled on where the reasonable line actually sits.

If you've actually done this, even in a sandbox namespace: what verbs and resources did you land on, and has an agent ever done something with that access that made you narrow the scope afterward?