Orchestrating Long-Running, Retryable Workflows with Temporal.io on Kubernetes: End-to-End Setup and Best Practices

Managing complex, long‑running, and retry‑enabled workflows in micro‑services architectures can be daunting. Temporal.io on Kubernetes solves this by providing a fault‑tolerant execution engine that scales with demand, guarantees at‑least‑once execution, and exposes a simple Go/Python SDK for rapid development.

Why Temporal.io is a Natural Fit for Long‑Running Workflows

Temporal’s core abstractions—workflows, activities, and signals—are engineered for precisely the problems you face when orchestrating stateful, retryable business processes. Unlike bare‑bone message queues, Temporal preserves execution state in a relational store, automatically replays events on worker restarts, and builds in programmable retry logic.

Prerequisites & Tool Stack

Cluster & Helm

• Kubernetes 1.23+ (IBM Cloud, GKE, EKS, or on‑prem)
• Helm 3.x; cert‑manager for TLS
• PostgreSQL 13+ as Temporal persistence (can use RDS or managed DB)
• Metrics: Prometheus, Grafana 9.x
• Logging: Loki or Fluentd
• Optional: Jaeger for distributed tracing

Local Development

Use docker compose with the official Temporal dev stack or k3d for a lightweight cluster.

Deploying Temporal on Kubernetes with Helm

1. Install cert‑manager: helm repo add jetstack https://charts.jetstack.io && helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.8.2
2. Add Temporal chart repo: helm repo add temporal https://temporal.io/helm && helm repo update
3. Configure values.yaml:
   • temporal.persistence.postgresql.enabled=true – enables sidecar DB; else provide external host, user, and password.
   • temporal.metrics.exporters.prometheus.enabled=true
   • worker.replicaCount=3 – start with three workers for baseline high availability.
   • searchAttributes.enabled=true
4. Install Temporal: helm install temporal temporal/temporal -f values.yaml --namespace temporal --create-namespace
5. Wait for ready state: kubectl rollout status deployment/temporal-frontend -n temporal

Post‑install, verify readiness with the Temporal Web UI (http://frontend-app:8080) and make sure the Prometheus scrape endpoint is working.

Configuring Persistence & Retention Policies

Temporal stores workflow execution history in Postgres. To balance durability and cost:

• Set workflowRetentionPeriod: 30 days for most business flows.
• Enable archivalEnabled in the cluster configuration to off‑load the history DB to S3 or Azure Blob and keep the source lightweight.
• Auto‑scale replicaCount on archival completion signals.

Scaling Workers with Kubernetes HPA

Workers execute activities; they can be auto‑scaled based on readiness and CPU usage.

1. Define a HorizontalPodAutoscaler on the worker deployment:
2. kubectl apply -f - <<EOF
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: temporal-worker-hpa
namespace: temporal
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: temporal-worker
minReplicas: 3
maxReplicas: 20
metrics:
- type: ContainerResource
containerResource:
name: cpu
target:
type: Utilization
averageUtilization: 60
EOF


For bursty workloads, consider adding keda scaler on queue length if your activities are backed by Kafka or SQS.

Implementing Retryable Workflows in Code

Temporal’s SDK lets you express retry policies declaratively.

// Go example
func FromFileToS3(ctx workflow.Context, fileURL string) error {
   ao := workflow.ActivityOptions{RetryPolicy: &workflow.RetryPolicy{
       InitialInterval:    time.Second * 5,
       BackoffCoefficient: 2.0,
       MaximumInterval:    time.Minute,
       MaximumAttempts:    10,
   }}
   ctx = workflow.WithActivityOptions(ctx, ao)
   err := workflow.ExecuteActivity(ctx, downloadFileActivity, fileURL).Get(ctx, nil)
   if err != nil { return err }
   err = workflow.ExecuteActivity(ctx, uploadS3Activity, fileURL).Get(ctx, nil)
   return err
}

Key points:

• Use idempotent activities to guarantee safe replay.
• Set maxAttempts based on SLA needs.
• Catch transient errors and return workflow.ErrNonRetryable when permanent failure is detected.

Monitoring & Observability

Temporal exposes metrics via Prometheus: temporal-frontend_serve_seconds_total, temporal-activityworker_requests_total, etc. Collect them with Grafana dashboards.

1. Deploy Prometheus via kube-prometheus-stack helm install prometheus oci://helmcharts.fqdn/prometheus --namespace monitoring
2. Import Temporal dashboard from public Grafana repo.
3. Set alerts: failure rate >5% over 5 min, workflow latency >1000ms.

For deeper tracing, integrate Jaeger and emit temporal-sdk-span tags from all activities.

Security & Compliance Considerations

• Enable mTLS between worker pods via cert‑manager certificates.
• Encrypt DB credentials with sealed‑secrets or Vault.
• Apply network policies to restrict traffic to Temporal services only from authorized namespaces.
• Audit the Web UI with OIDC integration (Keycloak or Auth0).

Troubleshooting Common Issues

1. Worker stuck at Running state: Check readinessProbe; bump memoryRequests if OOMKilled.
2. Long workflow manifestation: Verify workflowTimeout config; adjust retry policy or add compensating activities.
3. Database lock timeouts: Scale Postgres read replicas, partition workflow tables, or use Temporal archival.

Conclusion

By following this end‑to‑end process—installing Temporal on Kubernetes with Helm, configuring persistence, scaling workers automatically, coding robust retryable workflows, and embedding comprehensive observability—you can reliably run mission‑critical, long‑running operations at scale. Start small with a single team‑owned workflow, then expand to a multi‑tenant, sharded environment; Temporal’s architecture is designed for that evolution.

Ready to take the leap? Spin up a test cluster, clone the example repo from GitHub, and let Temporal transform your workflow infrastructure.