GitOps Best Practices for Teams

What is GitOps?

GitOps is a paradigm that uses Git repositories as the single source of truth for declarative infrastructure and applications. With Git at the center of your delivery pipelines, developers can make pull requests to accelerate and simplify application deployments and operations tasks to Kubernetes.

"GitOps is version-controlled infrastructure. It's treating your infrastructure the same way you treat your application code." — Weaveworks

Core GitOps Principles

• Declarative: The desired state of the system is expressed declaratively
• Versioned and Immutable: The desired state is stored in a way that enforces immutability and versioning (Git)
• Pulled Automatically: Software agents automatically pull the desired state from the source
• Continuously Reconciled: Agents continuously observe the actual system state and attempt to apply the desired state

GitOps vs Traditional CI/CD

While traditional CI/CD pipelines push changes to production, GitOps follows a pull-based model:

Traditional Push Model

1. Developer pushes code
2. CI builds and tests
3. CD pipeline pushes to production
4. Credentials needed in CI/CD system

GitOps Pull Model

1. Developer pushes code/config to Git
2. CI builds and pushes image
3. Config repo updated with new image tag
4. GitOps agent pulls changes to cluster
5. No credentials needed outside cluster

Popular GitOps Tools

Argo CD

Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. It's part of the CNCF graduated projects.

# Install Argo CD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Create an Application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/app-config
    targetRevision: HEAD
    path: kubernetes
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Flux

Flux is another CNCF project that keeps Kubernetes clusters in sync with configuration sources and automates updates when there's new code to deploy.

# Bootstrap Flux on a cluster
flux bootstrap github \
  --owner=my-org \
  --repository=fleet-infra \
  --path=clusters/production \
  --personal

# Define a GitRepository source
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: app-config
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/org/app-config
  ref:
    branch: main

Repository Structure Best Practices

Mono-repo vs Multi-repo

Choose the structure that fits your organization:

• Mono-repo: All configurations in one repository. Simpler to manage, single source of truth, but can become unwieldy at scale.
• Multi-repo: Separate repos for apps and infrastructure. Better access control, clearer ownership, but more complexity.

Recommended Directory Structure

# Multi-environment structure
├── base/                    # Common configurations
│   ├── deployment.yaml
│   ├── service.yaml
│   └── kustomization.yaml
├── overlays/
│   ├── development/
│   │   ├── kustomization.yaml
│   │   └── patches/
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   └── patches/
│   └── production/
│       ├── kustomization.yaml
│       └── patches/
└── helm-releases/           # Helm chart configurations
    ├── nginx.yaml
    └── cert-manager.yaml

Best Practice: Separate Application and Config Repos

Maintain separate repositories for application code and deployment configuration:

• App Repo: Contains application source code, Dockerfile, unit tests
• Config Repo: Contains Kubernetes manifests, Helm values, environment configs

This separation allows:

• Different permission models for devs and ops
• Independent versioning of apps and configs
• Clear audit trail of config changes
• Easier rollback without code changes

Implementing GitOps Workflows

Image Update Automation

Automate image tag updates in your config repo when new images are pushed:

# Flux ImageUpdateAutomation
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 1m
  sourceRef:
    kind: GitRepository
    name: app-config
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        name: fluxbot
        email: fluxbot@company.com
      messageTemplate: 'Update {{.ImageName}} to {{.NewTag}}'
    push:
      branch: main
  update:
    path: ./clusters/production
    strategy: Setters

Progressive Delivery

Implement canary deployments with GitOps using Argo Rollouts or Flagger:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: my-app
spec:
  replicas: 5
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {duration: 5m}
      - setWeight: 40
      - pause: {duration: 5m}
      - setWeight: 60
      - pause: {duration: 5m}
      - setWeight: 80
      - pause: {duration: 5m}
  selector:
    matchLabels:
      app: my-app
  template:
    # ... pod template

Secrets Management

Never store plain-text secrets in Git. Use one of these approaches:

Sealed Secrets

Encrypt secrets that can only be decrypted by the cluster:

# Create a sealed secret
kubeseal --format=yaml --cert=sealed-secrets.pem \
  < secret.yaml > sealed-secret.yaml

External Secrets Operator

Sync secrets from external providers (AWS Secrets Manager, HashiCorp Vault):

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: db-secret
  data:
  - secretKey: password
    remoteRef:
      key: production/database
      property: password

Observability & Debugging

• Use Argo CD's UI to visualize sync status and resource health
• Set up notifications for sync failures (Slack, email, PagerDuty)
• Implement health checks in your manifests
• Use kubectl diff to preview changes before merging

Team Collaboration Tips

1. Pull Request Reviews: All config changes go through PRs with approval requirements
2. Automated Testing: Run policy checks (OPA/Gatekeeper) and linting on PRs
3. Environment Promotion: Use Git branches or directory structures for env promotion
4. Documentation: Document your GitOps workflows and repo structure
5. Rollback Strategy: Define and practice rollback procedures

Conclusion

GitOps provides a powerful model for managing Kubernetes deployments with improved security, auditability, and developer experience. By treating your infrastructure as code stored in Git, you gain all the benefits of version control: history, rollbacks, collaboration, and review processes.

At VESTLABZ, we help teams adopt GitOps practices, set up the right tooling, and design workflows that scale with your organization. Whether you're starting your Kubernetes journey or optimizing existing deployments, our DevOps engineers can guide you through the transformation.