Golden Images Pipeline: Building Templates Like a Factory

Manual template creation works once. Install OS, configure, convert to template. Done. Until you need to update it. Then you do it again manually. And again. Eventually, no one remembers what’s in the template or how it was built.

A golden image pipeline treats templates like software: version controlled, automatically built, tested before use. When you need an update, you change the code and the pipeline builds a new template.

Images must be reproducible. If you can’t rebuild an identical image from code, you have a unique snowflake, not a template.

What Makes a Golden Image

A golden image is a VM template with:

  • Known contents: Every package, config, and file is documented
  • Reproducible build: Same inputs = same output
  • Versioned: v1, v2, v3 with change history
  • Tested: Verified before production use
  • Immutable: Never modified after creation

Manual Pipeline (Simple Start)

Before Packer, understand the process:

1. Download Cloud Image

Terminal window
# Ubuntu cloud image
wget https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
# Debian cloud image
wget https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-generic-amd64.qcow2
# AlmaLinux cloud image
wget https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/AlmaLinux-9-GenericCloud-latest.x86_64.qcow2

2. Create VM and Import

Terminal window
# Create VM
qm create 9000 --name "ubuntu-2404-base" --memory 2048 --cores 2 \
--net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-pci
# Import cloud image
qm importdisk 9000 noble-server-cloudimg-amd64.img local-zfs
# Attach disk
qm set 9000 --scsi0 local-zfs:vm-9000-disk-0
# Add cloud-init drive
qm set 9000 --ide2 local-zfs:cloudinit
# Boot settings
qm set 9000 --boot c --bootdisk scsi0
qm set 9000 --serial0 socket --vga serial0

3. Customize (Optional)

Terminal window
# Start VM with temporary cloud-init
qm set 9000 --ciuser temp --cipassword temp123
qm start 9000
# SSH in and customize
ssh temp@<ip>
sudo apt update && sudo apt upgrade -y
sudo apt install -y qemu-guest-agent vim htop curl git
# Clean up
sudo cloud-init clean
sudo rm -rf /var/log/*.log
sudo rm -rf /tmp/*
sudo shutdown -h now

4. Convert to Template

Terminal window
# Remove temporary cloud-init settings
qm set 9000 --delete ciuser,cipassword
# Convert to template
qm template 9000

5. Version and Document

Terminal window
# Rename with version
qm set 9000 --name "ubuntu-2404-v1"
# Add description
qm set 9000 --description "Ubuntu 24.04 LTS
Version: 1
Date: 2025-01-08
Base: noble-server-cloudimg-amd64.img
Packages: qemu-guest-agent, vim, htop, curl, git
Changes: Initial release"

Packer Pipeline (Automated)

Packer automates the entire process.

Install Packer

Terminal window
# HashiCorp repository
wget -O - https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
apt update && apt install packer
# Install Proxmox plugin
packer plugins install github.com/hashicorp/proxmox

Directory Structure

packer-templates/
├── templates/
│ ├── ubuntu-2404/
│ │ ├── ubuntu-2404.pkr.hcl
│ │ ├── variables.pkr.hcl
│ │ ├── http/
│ │ │ └── user-data
│ │ └── scripts/
│ │ ├── base.sh
│ │ ├── cleanup.sh
│ │ └── cloud-init.sh
│ └── debian-12/
│ └── ...
├── common/
│ ├── scripts/
│ │ └── common-packages.sh
│ └── files/
│ └── motd
└── Makefile

Packer Template

templates/ubuntu-2404/ubuntu-2404.pkr.hcl
packer {
required_plugins {
proxmox = {
version = ">= 1.1.0"
source = "github.com/hashicorp/proxmox"
}
}
}
source "proxmox-iso" "ubuntu-2404" {
# Proxmox connection
proxmox_url = var.proxmox_url
username = var.proxmox_username
token = var.proxmox_token
node = var.proxmox_node
insecure_skip_tls_verify = true
# VM settings
vm_id = var.vm_id
vm_name = "ubuntu-2404-v${var.version}"
# ISO (get checksum from releases.ubuntu.com)
iso_url = "https://releases.ubuntu.com/24.04/ubuntu-24.04-live-server-amd64.iso"
iso_checksum = "sha256:YOUR_CHECKSUM_HERE" # Get from SHA256SUMS file
iso_storage_pool = "local"
unmount_iso = true
# Hardware
cores = 2
memory = 2048
cpu_type = "host"
# Disk
scsi_controller = "virtio-scsi-pci"
disks {
disk_size = "32G"
storage_pool = var.storage_pool
type = "scsi"
}
# Network
network_adapters {
model = "virtio"
bridge = "vmbr0"
}
# Cloud-init
cloud_init = true
cloud_init_storage_pool = var.storage_pool
# Boot
boot_command = [
"<esc><wait>",
"e<wait>",
"<down><down><down><end>",
" autoinstall ds=nocloud-net;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/",
"<f10>"
]
boot_wait = "5s"
# HTTP server for autoinstall
http_directory = "http"
# SSH
ssh_username = "packer"
ssh_password = "packer"
ssh_timeout = "20m"
# Template
template_name = "ubuntu-2404-v${var.version}"
template_description = "Ubuntu 24.04 LTS - Built ${timestamp()}"
}
build {
sources = ["source.proxmox-iso.ubuntu-2404"]
# Base configuration
provisioner "shell" {
scripts = [
"scripts/base.sh",
"../../common/scripts/common-packages.sh"
]
}
# Copy files
provisioner "file" {
source = "../../common/files/motd"
destination = "/tmp/motd"
}
provisioner "shell" {
inline = ["sudo mv /tmp/motd /etc/motd"]
}
# Cleanup
provisioner "shell" {
scripts = [
"scripts/cleanup.sh",
"scripts/cloud-init.sh"
]
}
}

Variables

templates/ubuntu-2404/variables.pkr.hcl
variable "proxmox_url" {
type = string
default = "https://proxmox.lab.local:8006/api2/json"
}
variable "proxmox_username" {
type = string
default = "packer@pve!automation"
}
variable "proxmox_token" {
type = string
sensitive = true
}
variable "proxmox_node" {
type = string
default = "pve1"
}
variable "vm_id" {
type = number
default = 9000
}
variable "storage_pool" {
type = string
default = "local-zfs"
}
variable "version" {
type = string
default = "1"
}

Provisioning Scripts

scripts/base.sh
#!/bin/bash
set -ex
# Wait for cloud-init
cloud-init status --wait
# Update system
sudo apt update
sudo apt upgrade -y
# Install packages
sudo apt install -y \
qemu-guest-agent \
vim \
htop \
curl \
wget \
git \
jq \
unzip
# Enable guest agent
sudo systemctl enable qemu-guest-agent
scripts/cleanup.sh
#!/bin/bash
set -ex
# Clean apt
sudo apt autoremove -y
sudo apt clean
# Clean logs
sudo journalctl --rotate
sudo journalctl --vacuum-time=1s
sudo rm -rf /var/log/*.log
sudo rm -rf /var/log/*.gz
# Clean temp
sudo rm -rf /tmp/*
sudo rm -rf /var/tmp/*
# Clean SSH keys (regenerate on first boot)
sudo rm -f /etc/ssh/ssh_host_*
# Clean machine-id
sudo truncate -s 0 /etc/machine-id
sudo rm -f /var/lib/dbus/machine-id
# Clean history
history -c
sudo rm -f /root/.bash_history
sudo rm -f /home/*/.bash_history
scripts/cloud-init.sh
#!/bin/bash
set -ex
# Reset cloud-init
sudo cloud-init clean
# Remove network config (cloud-init will regenerate)
sudo rm -f /etc/netplan/*.yaml
# The template is now ready
echo "Template preparation complete"

Build Template

Terminal window
# Set token
export PKR_VAR_proxmox_token="xxxx-xxxx-xxxx"
# Validate
packer validate templates/ubuntu-2404/
# Build
packer build -var "version=2" templates/ubuntu-2404/

Cloud Image Pipeline (Faster)

Skip ISO install by starting with cloud images:

templates/ubuntu-2404-cloud/ubuntu-2404-cloud.pkr.hcl
source "proxmox-clone" "ubuntu-2404" {
# Clone from uploaded cloud image
clone_vm = "ubuntu-2404-cloud-base"
# ... rest of config
}

Pre-upload cloud image:

Terminal window
# Download and upload cloud image to Proxmox
wget https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
# Create base VM from cloud image (one-time)
qm create 8000 --name "ubuntu-2404-cloud-base" ...
qm importdisk 8000 noble-server-cloudimg-amd64.img local-zfs
qm set 8000 --scsi0 local-zfs:vm-8000-disk-0
qm template 8000

Packer then clones from base, customizes, creates versioned template.

CI/CD Integration

GitLab CI

.gitlab-ci.yml
stages:
- validate
- build
- test
variables:
PACKER_VERSION: "1.10.0"
validate:
stage: validate
script:
- packer init templates/ubuntu-2404/
- packer validate templates/ubuntu-2404/
build:
stage: build
script:
- packer build -var "version=${CI_PIPELINE_IID}" templates/ubuntu-2404/
only:
- main
test:
stage: test
script:
- ./scripts/test-template.sh ubuntu-2404-v${CI_PIPELINE_IID}
only:
- main

GitHub Actions

.github/workflows/build-template.yml
name: Build Template
on:
push:
branches: [main]
paths:
- 'templates/**'
jobs:
build:
runs-on: self-hosted # Need access to Proxmox
steps:
- uses: actions/checkout@v4
- name: Setup Packer
uses: hashicorp/setup-packer@main
with:
version: "1.10.0"
- name: Init Packer
run: packer init templates/ubuntu-2404/
- name: Build Template
env:
PKR_VAR_proxmox_token: ${{ secrets.PROXMOX_TOKEN }}
run: |
packer build -var "version=${{ github.run_number }}" templates/ubuntu-2404/

Testing Templates

Automated Test Script

scripts/test-template.sh
#!/bin/bash
TEMPLATE=$1
TEST_VM_ID=9999
echo "Testing template: ${TEMPLATE}"
# Clone template
qm clone $(qm list | grep "${TEMPLATE}" | awk '{print $1}') ${TEST_VM_ID} --name "template-test"
# Configure cloud-init
qm set ${TEST_VM_ID} --ciuser test --sshkeys ~/.ssh/id_ed25519.pub
qm set ${TEST_VM_ID} --ipconfig0 ip=dhcp
# Start VM
qm start ${TEST_VM_ID}
# Wait for boot
sleep 60
# Get IP
IP=$(qm guest cmd ${TEST_VM_ID} network-get-interfaces | jq -r '.[1]["ip-addresses"][0]["ip-address"]')
# Run tests
echo "Testing SSH..."
ssh -o StrictHostKeyChecking=no test@${IP} "echo 'SSH OK'"
echo "Testing packages..."
ssh test@${IP} "which vim htop curl git"
echo "Testing guest agent..."
qm agent ${TEST_VM_ID} ping
# Cleanup
qm stop ${TEST_VM_ID}
qm destroy ${TEST_VM_ID}
echo "All tests passed!"

Test Checklist

[ ] VM boots successfully
[ ] Cloud-init completes
[ ] SSH works with key auth
[ ] Required packages installed
[ ] Guest agent responds
[ ] No leftover sensitive data
[ ] Machine-id regenerated
[ ] SSH host keys regenerated

Versioning Strategy

Semantic Versioning

ubuntu-2404-v1.0.0 # Major: Breaking changes
ubuntu-2404-v1.1.0 # Minor: New features
ubuntu-2404-v1.1.1 # Patch: Bug fixes

Date-Based Versioning

ubuntu-2404-20250108 # Date of build
ubuntu-2404-202501 # Month of build

Build Number

ubuntu-2404-b42 # CI build number

Tracking Active Version

Terminal window
# Create symlink-style alias
qm set 9002 --description "... LATEST: true"
# Or use tags
qm set 9002 --tags "latest,ubuntu,2404"

The Lesson

Images must be reproducible. Otherwise, they’re unique snowflakes.

A template you clicked together manually is:

  • Undocumented (what’s in it?)
  • Unreproducible (can you rebuild it exactly?)
  • Untested (does it actually work?)
  • Unversioned (which version is this?)

A golden image pipeline produces templates that are:

  • Documented (code shows everything)
  • Reproducible (same code = same image)
  • Tested (automated tests before use)
  • Versioned (clear history of changes)

The investment in automation pays off when:

  • Security update needed → rebuild all templates
  • New requirement → change code, rebuild
  • “What’s in this template?” → read the code
  • Audit requirement → show build history

Start simple: document your manual process. Then automate it. Then add CI/CD. Each step makes your templates more reliable and less mysterious.