Airgap OpenShift Installation: move the registry created using oc adm release mirror between environments

Some customers, especially large banks, have very tight security requirements. Most of them enforce a complete disconnection of their internal networks from the Internet.

When installing OpenShift in such environments (this is named “disconnected” or “airgap” installation), all the OpenShift images have to be fetched (thanks to oc adm release mirror) in a dedicated registry from a bastion host that is both on the internal network and on the Internet.

However, for some customers this is not secure enough. Most of them would rather download all the images locally (using oc adm release mirror ?), transport them on a removable media to the internal network and provision the target registry.

As described in this article, skopeo and Ansible can be a nice complement of oc adm release mirror to achieve this setup. Let’s discover how!

The rest of this guide assumes that you followed the official documentation and fetched on the bastion node all the required images in a dedicated registry using oc adm release mirror.

First, you will need a token that has administrative privileges on this registry.

$ oc whoami -t
AZERTYUIOPQSDFGHJKLMWXCVBN1234567890azertyu

Store it somewhere for later use.

export TOKEN=$(oc whoami -t)

Confirm your token can fetch the registry catalog.

$ curl -s https://docker-registry.default.svc:5000/v2/_catalog -H "Authorization: Bearer $TOKEN" |jq .

{
  "repositories": [
    "openshift/httpd",
    "openshift/java",
    "openshift/jenkins",
    [...]
    "openshift/php",
    "openshift/python"
  ]
}

As you may have guessed, this is the list of all the images provisioned by the oc adm release mirror command that we will need to export using skopeo. But before doing so, we need to get the list of all the tags of each image.

Hopefully, there is also an API for this.

$ curl -s https://docker-registry.default.svc:5000/v2/openshift/php/tags/list -H "Authorization: Bearer $TOKEN" |jq .

{
  "name": "openshift/php",
  "tags": [
    "latest",
    "5.5",
    "5.6",
    "7.0",
    "7.1"
  ]
}

As an example, to export openshift/php:5.5 from the docker-registry.default.svc:5000 registry to the local filesystem (in /tmp/oci_registry), you could use:

skopeo --insecure-policy copy --src-tls-verify=false --src-creds=admin:$TOKEN docker://docker-registry-default.app.itix.fr/openshift/php:5.5 oci:/tmp/oci_registry:openshift/php:5.5

We now have the basis to build the Ansible playbook that will dump the registry created by oc adm release mirror to the filesystem.

The first step of this playbook would be to fetch the registry catalog. There is nothing fancy here, just a plain application of the uri module.

- hosts: localhost
  gather_facts: no
  vars:
    registry: docker-registry.default.svc:5000
    validate_certs: false
  tasks:
  - name: Fetch the catalog of the docker registry
    uri:
      url: 'https://{{ registry }}/v2/_catalog'
      headers:
        Authorization: Bearer {{ token }}
      status_code: 200
      return_content: yes
      validate_certs: '{{ validate_certs }}'
    register: catalog

The next step is to iterate on this catalog to fetch the tags of each image. The url lookup plugin is used to query the registry API. The image name is added in front of each tag (to construct the full image name: image:tag) using the cartesian product filter.

  - name: Construct a list of all available images
    set_fact:
      images: >
        {{ images|default([]) + new_images }}        
    vars:
      image_tags: >
        {{ (lookup("url", "https://"~registry~"/v2/"~item~"/tags/list",
                          headers={"Authorization": "Bearer "~token},
                          validate_certs=validate_certs)|from_json).tags }}        
      new_images: >
        {{ [item] | product(image_tags) | map('join', ':') | list }}        
    loop: '{{ catalog.json.repositories }}'

  - debug:
      var: images

When using the url lookup plugin on MacOS, you might need to set the OBJC_DISABLE_INITIALIZE_FORK_SAFETY environment variable as explained in #32499.

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

Finally, skopeo is called to download each image to /tmp/oci_registry.

- hosts: localhost
  gather_facts: no
  vars:
    target: /tmp/oci_registry
    [...]

  tasks:

  [...]

  - name: Downloading OpenShift images...
    command: skopeo --insecure-policy copy --src-tls-verify={{ validate_certs|bool|ternary('true','false') }} --src-creds=admin:{{ token }} 'docker://{{ registry }}/{{ item }}' 'oci:{{ target }}:{{ item }}'
    with_items: '{{ images }}'

The complete playbook is available here and can be run as follow.

ansible-playbook pull.yaml -e token=$TOKEN

This will dump the registry at docker-registry.default.svc:5000 to /tmp/oci_registry. If you want to target another registry or store the images somewhere else, you can pass the registry or target extra variables.

ansible-playbook pull.yaml -e token=$TOKEN -e registry=docker-registry-default.app.openshift.test -e target=/tmp/oci_registry

The images are stored as an OCI registry whose format is standardized. It can be moved somewhere else, using a removable media for instance.

How this OCI registry can be imported in the target registry is a subject for another article.

Stay tuned.