Provide a way to reproduce our registry for air-gapped users

[phorcys420]

We currently lack a proper way for customers to reproduce 1:1 our currently published modules on registry.coder.com. While they could rely on the main branch, we have unpublished changes.

@bcpeinhardt suggested that we could do it the following ways:

• Publish the assets directory (every version of every modules in a .tar.gz file) that coder/registry-server provides as an artifact
• Publish the docker image with the registry server in it so customers could just run it themselves

I think no. 2 is the better option, but it also takes some more time and might generate more support needs.
Meanwhile, no. 1 is the fastest to get rolling so we can provide air-gapped users with something in an "it works" state, and then look into getting the work done to publish the registry server.

cc: @matifali @bpmct for input

[matifali]

A git clone of the current repo, and using source in the format below should work. more docs ate: https://developer.hashicorp.com/terraform/language/modules/sources#generic-git-repository

source = "git::https://git.example.com/internal/registry.git//registry/coder/modules/module_name"

Although they won't be able to use versioning with git sources.

[jad1010]

Hi, if possible, can it be done in a way that it maintains all module versions and can be set up as a remote repository in artifactory. We currently use remote repository to proxy/download/cache things, then move those to another airgapped artifactory instance.

We have been able to do this with the coder provider.

[DevelopmentCats]

I was reading the thread in discord for this, and I was personally thinking that #1 would be a good idea before I even saw this.

[matifali]

Hi, if possible, can it be done in a way that it maintains all module versions and can be set up as a remote repository in artifactory. We currently use remote repository to proxy/download/cache things, then move those to another airgapped artifactory instance.

According to Terraform:

Version constraints are supported only for modules installed from a module registry, such as the public Terraform Registry or HCP Terraform's private module registry. Other module sources can provide their own versioning mechanisms within the source string itself, or might not support versions at all. In particular, modules sourced from local file paths do not support version; since they're loaded from the same source repository, they always share the same version as their caller.

[jad1010]

Hi, if possible, can it be done in a way that it maintains all module versions and can be set up as a remote repository in artifactory. We currently use remote repository to proxy/download/cache things, then move those to another airgapped artifactory instance.

According to Terraform:

Version constraints are supported only for modules installed from a module registry, such as the public Terraform Registry or HCP Terraform's private module registry. Other module sources can provide their own versioning mechanisms within the source string itself, or might not support versions at all. In particular, modules sourced from local file paths do not support version; since they're loaded from the same source repository, they always share the same version as their caller.

I think it would be better to have the structure setup like this https://github.com/terraform-aws-modules where each module is a project with a release and tag etc. then it would be easier to pull those in. With the old repo setup where everything was the same version it was easy to just clone and publish with jf cli "jf tf p".

maybe if it was setup like the modules org above then something like this could be done.

  required_providers {
    artifactory = { source = "jfrog/artifactory", version = "~>3.0" }
    http         = { source = "hashicorp/http",         version = "~>3.0" }
  }
}

provider "artifactory" {
  url      = var.artifactory_url
  username = var.username
  password = var.password
}

data "http" "gh_repos" {
  url = "https://api.github.com/orgs/terraform-aws-modules/repos?per_page=100"
  request_headers = {
    Authorization = "token ${var.github_token}"
    Accept        = "application/vnd.github.v3+json"
  }
}

locals {
  modules = [for r in jsondecode(data.http.gh_repos.body) : r.name]
}

resource "artifactory_remote_vcs_repository" "mod" {
  for_each    = toset(local.modules)
  key         = "terraform-aws-${each.key}-vcs"
  url         = "https://github.com/terraform-aws-modules/${each.key}.git"
  package_type = "VCS"

  vcs {
    vcs_type = "GIT"
    github {
      enabled = true
    }
  }
}

set -euo pipefail

# Configuration
ART_URL="https://<ART_URL>/artifactory"
ART_MODULE_REPO="terraform-modules-local"
PROVIDER="aws"
EXCLUSIONS="*test*;*.md"
MAX_RETRIES=5
RETRY_DELAY=15
PARALLEL_JOBS=4
DRY_RUN=false

if [[ "${1:-}" == "--dry-run" ]]; then
  DRY_RUN=true
  echo "[DRY-RUN] No actual changes will be made."
fi

LOGFILE="publish-$(date +%Y%m%d_%H%M%S).log"
exec > >(tee -a "$LOGFILE") 2>&1
echo "Started: $(date)"

retry() {
  local n=1
  until "$@" ; do
    if (( n++ < MAX_RETRIES )); then
      echo "Retry $n/$MAX_RETRIES in $RETRY_DELAY s..."
      sleep "$RETRY_DELAY"
    else
      echo "❌ Command failed after $MAX_RETRIES attempts."
      return 1
    fi
  done
}

run_cmd() {
  if $DRY_RUN; then
    echo "[DRY] $*"
  else
    eval "$@"
  fi
}

process_module() {
  mod=$1
  echo -e "\n➡ Processing module: $mod"

  git_url="$ART_URL/vcs/terraform-aws-${mod}-vcs"
  tmp=$(mktemp -d)
  run_cmd "git clone --mirror \"$git_url\" \"$tmp/$mod.git\"" || { echo "Clone failed"; return; }
  cd "$tmp/$mod.git"
  TAG=$(run_cmd "git tag" | tr ' ' '\n' | sort -V | tail -n1 || true)
  if [[ -z "$TAG" ]]; then echo "No tags found; skipping."; return; fi
  VERSION=${TAG#v}
  echo "Using tag $TAG → version $VERSION"

  cnt=$(run_cmd "jfrog rt s --count \"$ART_MODULE_REPO/terraform-aws-modules/$mod/$PROVIDER/$VERSION/*\" --fail-no-op")
  if [[ "$cnt" != "0" ]]; then
    echo "✅ Already published; skipping."
    return
  fi

  git clone --branch "$TAG" "$tmp/$mod.git" "$tmp/$mod"
  pub_cmd="jfrog rt tfp \"$tmp/$mod/\" \"$ART_MODULE_REPO\" \
    --namespace terraform-aws-modules --module-name \"$mod\" \
    --provider \"$PROVIDER\" --module-version \"$VERSION\" \
    --exclusions \"$EXCLUSIONS\""
  echo "📤 Publishing $mod v$VERSION"
  run_cmd retry bash -c "$pub_cmd" || echo "Publish failed for $mod"
}

export ART_URL ART_MODULE_REPO PROVIDER EXCLUSIONS DRY_RUN retry run_cmd process_module

# Discover all VCS repos dynamically
mods=$(curl -s -u "user:password" "$ART_URL/api/repositories?type=remote" |
  jq -r '.[].key | select(startswith("terraform-aws-") and endswith("-vcs")) |
         sub("^terraform-aws-";"") | sub("-vcs$";"")')

echo "$mods" | parallel -j "$PARALLEL_JOBS" process_module {}

echo "Finished: $(date). Log: $LOGFILE"

[matifali]

Nice. But it would become overly complex to manage if we chose a different repo for each of the modules. I would think more about the best way to clone the registry in Artifactory and still provide versioning.

[jad1010]

Nice. But it would become overly complex to manage if we chose a different repo for each of the modules. I would think more about the best way to clone the registry in Artifactory and still provide versioning.

Yeah, but how are the versions pushed? Last I looked, there are modules on the registry site that are not even tagged in the repository. Maybe they are still from the old repo. I could not figure out how to keep the versions. That was sort of the point of the whole issue.

[matifali]

Maybe they are still from the old repo

Yes

I could not figure out how to keep the versions. That was sort of the point of the whole issue.

We manage versioning in internal repositories that host the registry server's code based on the tag in this repository and the older coder/modules repository. In the future, we can open-source it for users to deploy in their infrastructure.

cc: @bcpeinhardt for your thoughts on how best to help our users here. TLDR: Users want an easy way to replicate our registry locally in Artifactory or an alternative solution.