Posted in

OpenShift: Operator Management with oc-mirror v2 & podman

johngohby johngoh••0

Faster Operator and Channel Information Extraction for Disconnected Environments

OpenShift 4.19 introduces significant improvements to the oc-mirror v2 tool, offering administrators a more efficient way to manage operators and extract channel information in disconnected and air-gapped environments. This enhanced version represents a major leap forward in simplifying the complex process of mirroring container images and operator catalogs.

What is oc-mirror v2?

The oc-mirror v2 is a command-line tool designed to mirror OpenShift Container Platform (OCP) releases, operator catalogs, and additional images. It serves as a lifecycle manager for internet-restricted environments, providing three distinct workflow options:

  • mirrorToDisk (m2d): Pulls container images from specified sources and packages them into tar archives on local storage
  • diskToMirror (d2m): Transfers container images from tar archives to container registries
  • mirrorToMirror (m2m): Directly copies container images from source to destination registries

Key Improvements in Version 2

The v2 iteration addresses several critical challenges that administrators faced with the previous version. Most notably, it provides faster extraction of operator and channel information, significantly reducing the time required to process large operator catalogs in enterprise environments.

Workflow Optimization for Disconnected Environments

One of the most significant improvements is the tool’s ability to handle custom operator catalogs more efficiently. For organizations operating in air-gapped environments, oc-mirror v2 provides a more streamlined approach to operator management. The tool generates appropriate manifests including:

  • ImageDigestMirrorSets & ImageTagMirrorSets configurations
  • CatalogSource definitions
  • Update definitions for deployment across clusters

Getting Started

To begin using oc-mirror v2, administrators need:

  • Cluster administrator privileges on an installed OpenShift cluster
  • The oc-mirror binary downloaded from Red Hat’s official distribution
  • Access to a container registry for publishing mirrored content
  • Podman for container operations

Basic Workflow Commands

Mirror-to-Disk (m2d) – For environments with internet access:

oc-mirror --config isc.yaml file://<directory_name> --v2

Disk-to-Mirror (d2m) – For transferring to disconnected registries:

oc-mirror --config isc.yaml --from file://<directory_name> docker://<remote_registry> --v2

Mirror-to-Mirror (m2m) – For direct registry-to-registry mirroring:

oc-mirror --config isc.yaml --workspace file://<directory_name> docker://<remote_registry> --v2

Operator-Specific Commands

List Available Operators:

oc-mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.19
oc-mirror list operators --catalog=registry.redhat.io/redhat/certified-operator-index:v4.19
oc-mirror list operators --catalog=registry.redhat.io/redhat/community-operator-index:v4.19
oc-mirror list operators --catalog=registry.redhat.io/redhat/redhat-marketplace-index:v4.19

# Install opm if not already available
curl -L https://github.com/operator-framework/operator-registry/releases/latest/download/linux-amd64-opm -o opm
chmod +x opm
sudo mv opm /usr/local/bin/

# Pull the index image first
podman pull registry.redhat.io/redhat/redhat-operator-index:v4.19

# Use opm to get operators with their channels
opm render registry.redhat.io/redhat/redhat-operator-index:v4.19 | \
  jq -r 'select(.schema == "olm.channel") | "\(.package):\(.name)"' | \
  sort

# Or get all channel data in structured format
opm render registry.redhat.io/redhat/redhat-operator-index:v4.19 | jq 'select(.schema == "olm.channel") | {package: .package, channel: .name, entries: .entries}'

# List all operators in the index
# simple opm list all operators without channel.
opm render registry.redhat.io/redhat/redhat-operator-index:v4.19 | jq -r 'select(.schema == "olm.package") | .name' | sort | uniq

Generated Resources Location

After successful mirroring, cluster resources are generated in:

<workspace_path>/working-dir/cluster-resources

These include ImageDigestMirrorSet (IDMS), CatalogSource, and ClusterCatalog manifests that need to be applied to the target OpenShift cluster.

Debugging

Check node uses the right registry

oc debug node/master-2 -- chroot /host podman pull registry.us.kubernetes.day/quayadmin/redhat/certified-operator-index:v4.18

# Check registries.conf again
oc debug node/master-2 -- chroot /host cat /etc/containers/registries.conf

# Check for any network policies affecting the runtime
oc debug node/master-2 -- chroot /host systemctl status crio

Check that the imagedigests are updated,

oc edit ImageDigestMirrorSet/image-digest-mirror

# add broader mapping

# idms-support-tools.yaml
# Mirror by DIGEST (authoritative) for entire registries
apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
  name: mirror-quay-and-registry-redhat
spec:
  imageDigestMirrors:
  - source: registry.redhat.io
    mirrors:
    - registry.kubernetes.day/ocp419
  - source: quay.io
    mirrors:
    - registry.kubernetes.day/ocp419
---
# itms-support-tools.yaml (optional but handy for tag-based pulls)
# Optional: also mirror TAG lookups (handy while you’re stabilizing)
apiVersion: config.openshift.io/v1
kind: ImageTagMirrorSet
metadata:
  name: mirror-quay-and-registry-redhat
spec:
  imageTagMirrors:
  - source: registry.redhat.io
    mirrors:
    - registry.kubernetes.day/ocp419
  - source: quay.io
    mirrors:
    - registry.kubernetes.day/ocp419

Mirror –v2 output

2025/08/10 17:33:21  [INFO]   : === Results ===
2025/08/10 17:33:21  [INFO]   :  ✓  191 / 191 release images mirrored successfully
2025/08/10 17:33:21  [INFO]   :  ✗  320 / 322 operator images mirrored: Some operator images failed to be mirrored - please check the logs
2025/08/10 17:33:21  [INFO]   :  ✓  1 / 1 additional images mirrored successfully
2025/08/10 17:33:21  [ERROR]  : [Worker] error mirroring image docker://registry.redhat.io/openshift-service-mesh/istio-cni-rhel9@sha256:54cada48e5c9824f255f82daa2ef5bea236919e521d3ea49885f2883ced2b7bc (Operator bundles: [servicemeshoperator3.v3.1.0] - Operators: [servicemeshoperator3]) error: copying image 1/4 from manifest list: writing blob: initiating layer upload to /v2/quayadmin/openshift-service-mesh/istio-cni-rhel9/blobs/uploads/ in registry.us.kubernetes.day: unauthorized: access to the requested resource is not authorized
2025/08/10 17:33:21  [ERROR]  : [Worker] error mirroring image docker://registry.redhat.io/openshift-service-mesh/istio-sail-operator-bundle@sha256:f50498d866f239446e5c9e01ee87e465ad920410ab385f89ebab3fd40dba9bb5 error: skipping operator bundle docker://registry.redhat.io/openshift-service-mesh/istio-sail-operator-bundle@sha256:f50498d866f239446e5c9e01ee87e465ad920410ab385f89ebab3fd40dba9bb5 because one of its related images failed to mirror
2025/08/10 17:33:21  [INFO]   : 📄 Generating IDMS file...
2025/08/10 17:33:21  [INFO]   : /data/ocmirror419/working-dir/cluster-resources/idms-oc-mirror.yaml file created
2025/08/10 17:33:21  [INFO]   : 📄 Generating ITMS file...
2025/08/10 17:33:21  [INFO]   : /data/ocmirror419/working-dir/cluster-resources/itms-oc-mirror.yaml file created
2025/08/10 17:33:21  [INFO]   : 📄 Generating CatalogSource file...
2025/08/10 17:33:21  [INFO]   : /data/ocmirror419/working-dir/cluster-resources/cs-certified-operator-index-v4-19.yaml file created
2025/08/10 17:33:21  [INFO]   : /data/ocmirror419/working-dir/cluster-resources/cs-redhat-operator-index-v4-19.yaml file created
2025/08/10 17:33:21  [INFO]   : 📄 Generating ClusterCatalog file...
2025/08/10 17:33:21  [INFO]   : /data/ocmirror419/working-dir/cluster-resources/cc-certified-operator-index-v4-19.yaml file created
2025/08/10 17:33:21  [INFO]   : /data/ocmirror419/working-dir/cluster-resources/cc-redhat-operator-index-v4-19.yaml file created
2025/08/10 17:33:21  [INFO]   : 📄 Generating Signature Configmap...
2025/08/10 17:33:21  [INFO]   : /data/ocmirror419/working-dir/cluster-resources/signature-configmap.json file created
2025/08/10 17:33:21  [INFO]   : /data/ocmirror419/working-dir/cluster-resources/signature-configmap.yaml file created
2025/08/10 17:33:21  [INFO]   : 📄 Generating UpdateService file...
2025/08/10 17:33:21  [INFO]   : /data/ocmirror419/working-dir/cluster-resources/updateService.yaml file created
2025/08/10 17:33:21  [INFO]   : mirror time     : 16m19.171112015s
2025/08/10 17:33:21  [INFO]   : 👋 Goodbye, thank you for using oc-mirror
2025/08/10 17:33:21  [ERROR]  : [Executor] [Worker] some errors occurred during the mirroring.
         Please review /data/ocmirror419/working-dir/logs/mirroring_errors_20250810_173321.txt for a list of mirroring errors.
         You may consider:
         * removing images or operators that cause the error from the image set config, and retrying
         * keeping the image set config (images are mandatory for you), and retrying
         * mirroring the failing images manually, if retries also fail.

Summary

The tool’s evolution from v1 to v2 demonstrates Red Hat’s commitment to addressing real-world challenges faced by enterprise customers, particularly those operating in security-conscious or air-gapped environments where efficient operator management is crucial for maintaining operational effectiveness.

Common Operators 
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
mirror:
  platform:
    channels:
      - name: stable-4.19
        type: ocp
        #minVersion: 4.19.6 # optional
        #maxVersion: 4.19.6 # optional
    graph: true
  operators:
    # Red Hat Operators Catalog
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.18
      packages:

      # Advanced Cluster Management for Kubernetes
      - name: advanced-cluster-management
      - name: multicluster-engine

      # Node Feature Discovery Operatr
      - name: nfd

      # Advanced Cluster Security for Kubernetes
      - name: rhacs-operator
        channels:
          - name: stable

      # Quay
      - name: quay-operator
      - name: quay-bridge-operator
      - name: container-security-operator

      # Data Foundation
      - name: odf-operator
      - name: odf-operator-controller-manager
      - name: odf-csi-addons-operator
      - name: ocs-operator
      - name: mcg-operator
      - name: odf-dependencies
      - name: rook-ceph-operator
      - name: recipe
      - name: cephcsi-operator
      - name: ocs-client-operator
      - name: odf-prometheus-operator
      - name: odr-cluster-operator
      - name: odr-hub-operator
      - name: odf-multicluster-orchestrator

      # Local Storage Operator
      - name: lvms-operator
      - name: local-storage-operator

      # Dev Tooling
      - name: web-terminal
      - name: devworkspace-operator
      - name: devspaces

      # NMState Operator
      - name: kubernates-nmstate-operator
        channels:
          - name: stable

      # Update Service
      - name: cincinnati-operator
        channels:
          - name: v1

      # Pipelines
      - name: openshift-pipelines-operator-rh
        channels:
          - name: latest

      # GitOps
      - name: openshift-gitops-operator
        channels:
          - name: latest

      # Virt
      - name: kubevirt-hyperconverged
        channels:
          - name: stable
      - name: mtv-operator

      # Compliance Operator
      - name: compliance-operator
        channels:
          - name: stable

      # File Integrity Operator
      - name: file-integrity-operator
        channels:
          - name: stable

      # Ansible Automation Platform Operator
      - name: ansible-automation-platform-operator
        channels:
          - name: stable-2.5

      # MetalLB Operator
      - name: metallb-operator
        channels:
          - name: stable

      # PTP Operator
      - name: ptp-operator
        channels:
          - name: stable

      # Openshift Logging
      - name: cluster-logging
        channels:
          - name: stable-6.3
      - name: loki-operator
        channels:
          - name: stable-6.3

      # Cluster Observablility Operator
      - name: cluster-observability-operator
        channels:
          - name: stable

      # Network Observablility Operator
      - name: netobserv-operator
        channels:
          - name: stable

      # Keycloak Operator
      - name: rhbk-operator

      # Cert Manager
      - name: openshift-cert-manager-operator

    # Certified Operators Catalog
    - catalog: registry.redhat.io/redhat/certified-operator-index:v4.18
      packages:

      # Nginx Operator
      - name: nginx-ingress-operator
        channels:
          - name: alpha

      # Nutanix CSI Operator
      - name: nutanixcsioperator
        channels:
          - name: stable-3.x

      # Gitlab Operator
      - name: gitlab-operator-kubernetes
        channels:
          - name: stable
      - name: gitlab-runner-operator
        channels:
          - name: stable

    # Community Operators Catalog
    - catalog: registry.redhat.io/redhat/community-operator-index:v4.18
      packages:

      # MariaDB Operator
      - name: mariadb-operator

      # GroupSync Operator
      - name: group-sync-operator
Common Additional Images 
  additionalImages:
  # Some standard tooling for debugging
  - name: registry.redhat.io/rhel9/support-tools:latest
  - name: registry.redhat.io/rhel8/rhel-guest-image:latest
  - name: registry.redhat.io/rhel9/rhel-guest-image:latest
  - name: registry.redhat.io/openshift4/ose-must-gather:latest
  - name: registry.redhat.io/ubi8/ubi:latest
  - name: registry.redhat.io/ubi9/ubi:latest
  - name: registry.redhat.io/ubi9/httpd-24:latest
  - name: registry.redhat.io/odf4/odf-must-gather-rhel9:v4.18
  - name: registry.redhat.io/container-native-virtualization/cnv-must-gather-rhel9:v4.19
  - name: registry.redhat.io/ubi9/httpd-24:latest
  - name: registry.redhat.io/openshift4/ose-cli:latest

  # Postgres Dataabse Images
  - name: registry.redhat.io/rhel9/postgresql-15:latest
  - name: registry.redhat.io/rhel9/postgresql-16:latest

Leave a Reply

Your email address will not be published. Required fields are marked *