Architecture Diagrams

📋 Installation Flow

• Step 1 - Bastion Setup: Configure dnsmasq (DNS/DHCP/TFTP), httpd, PXE boot files, create ignition file, download RHCOS images
• Step 2 - SNO Installation: Network boot SNO LPAR via PXE → DHCP assigns IP → PXE loads kernel & initramfs → Downloads rootfs from HTTP → Applies ignition configuration → Installs OpenShift
• Step 3 - Monitoring: Use openshift-install wait-for bootstrap-complete and wait-for install-complete, then verify with oc get nodes and oc get co

🔧 Key Services on Bastion LPAR

• dnsmasq: Provides DNS, DHCP, and TFTP services for PXE boot
• httpd: HTTP server (port 8000) serving ignition files and RHCOS images
• grub2: Network boot configuration for PowerVM

📁 Important File Locations

• Configuration: /etc/dnsmasq.conf, /etc/dnsmasq.d/addnhosts
• PXE Boot: /var/lib/tftpboot/boot/grub2/grub.cfg, /var/lib/tftpboot/rhcos/
• HTTP Content: /var/www/html/ignition/, /var/www/html/install/

⚖️ HAProxy Load Balancer Configuration

• API Server (6443): Load balances OpenShift API requests across all master nodes
• Machine Config Server (22623): Routes machine configuration requests during Day 1 installation
• Machine Config Server Day 2 (22624): Routes machine configuration requests for Day 2 worker additions
• HTTP Ingress (80): Load balances HTTP traffic to worker nodes (or masters if no workers)
• HTTPS Ingress (443): Load balances HTTPS traffic to worker nodes (or masters if no workers)
• Stats Dashboard (9000): HAProxy statistics and health monitoring interface

📋 Day 2 Installation Flow

• Step 1 - Service Setup: Update bastion services (dnsmasq, HAProxy) with new worker node information
• Step 2 - Create Ignition: Generate Day 2 ignition files for new worker nodes using existing cluster credentials
• Step 3 - Network Boot: PXE boot new worker LPARs using lpar_netboot command
• Step 4 - Monitor: Watch worker nodes join the cluster and approve CSRs (Certificate Signing Requests)

🔧 Complete Bastion Services for HA Clusters

• dnsmasq: DNS, DHCP, and TFTP services
• httpd: HTTP server for ignition files and RHCOS images
• HAProxy: Load balancer for API, machine config, and ingress traffic
• NFS (optional): Network storage for persistent volumes and image registry
• Squid (optional): HTTP proxy for restricted network environments
• Chrony (optional): Time synchronization service

✅ Benefits of Dedicated Bastion per Cluster

• 🔒 Security Isolation: Each cluster has its own isolated bastion, preventing cross-cluster security breaches. If one bastion is compromised, other clusters remain protected.
• 🎯 Independent Lifecycle Management: Upgrade, patch, or maintain each bastion independently without affecting other clusters. Different clusters can run different OpenShift versions.
• ⚡ Performance & Scalability: Dedicated resources per cluster prevent resource contention. Each bastion can be sized appropriately for its cluster's needs.
• 🛠️ Configuration Flexibility: Each bastion can have unique configurations (DNS zones, DHCP ranges, HAProxy rules) tailored to its cluster's requirements.
• 🔄 Simplified Troubleshooting: Issues are isolated to specific clusters. Logs, services, and configurations are separate, making debugging easier.
• 📊 Resource Accountability: Clear resource allocation and cost tracking per cluster. Each environment (prod, dev, test, DR) has dedicated infrastructure.
• 🚀 Parallel Operations: Perform Day 2 operations (adding workers, upgrades) on multiple clusters simultaneously without conflicts.
• 🔐 Network Segmentation: Each cluster can operate in different network zones with appropriate firewall rules and access controls.

🏗️ Typical Multi-Cluster Use Cases

• Production Cluster (Blue): Mission-critical workloads with high availability, strict SLAs, and enhanced monitoring
• Development Cluster (Green): Active development environment with frequent deployments and testing
• Testing/QA Cluster (Yellow): Pre-production validation, integration testing, and quality assurance
• Disaster Recovery Cluster (Red): Standby cluster in different location for business continuity

📋 Bastion Services per Cluster

• dnsmasq: Cluster-specific DNS zones (e.g., prod.ocp.io, dev.ocp.io, test.ocp.io, dr.ocp.io)
• httpd: Serves ignition files and RHCOS images for its cluster nodes
• HAProxy: Load balances API and ingress traffic for its cluster only
• NFS: Provides storage for its cluster's persistent volumes and image registry
• Monitoring: Each bastion can run cluster-specific monitoring and alerting

⚠️ Alternative: Shared Bastion Considerations

While a shared bastion for multiple clusters is possible, it introduces:

• ❌ Single point of failure affecting all clusters
• ❌ Complex configuration management with potential for conflicts
• ❌ Resource contention during simultaneous operations
• ❌ Reduced security isolation between environments
• ❌ Difficult troubleshooting with mixed logs and services

Recommendation: Use dedicated bastions for production and DR clusters at minimum. Dev/Test clusters may share a bastion if budget constraints exist.

❌ Approach 1: Corporate DNS Only - Challenges

• Slow Change Management:
  • ❌ DNS changes require tickets and approvals (days/weeks)
  • ❌ Each cluster creation/destruction needs DNS team involvement
  • ❌ Dev/test clusters can't be created on-demand
  • ❌ Troubleshooting requires waiting for DNS changes
• No Load Balancer:
  • ❌ Corporate LB provisioning is expensive and slow
  • ❌ May not support OpenShift-specific requirements
  • ❌ Overkill for dev/test clusters
• PXE Boot Issues:
  • ❌ Corporate DHCP can't provide PowerVM-specific boot parameters
  • ❌ No integration with TFTP for netboot
  • ❌ Can't serve ignition files during boot
• Operational Overhead:
  • ❌ Every cluster operation requires coordination with DNS team
  • ❌ Day 2 operations (adding workers) need DNS updates
  • ❌ Can't quickly iterate during troubleshooting

✅ Approach 2: Bastion DNS - Benefits

• Self-Service & Speed:
  • ✅ Create/destroy clusters instantly without DNS tickets
  • ✅ Modify DNS entries immediately during troubleshooting
  • ✅ Dev/test clusters can be spun up on-demand
  • ✅ Day 2 operations (add workers) are instant
• Integrated Services:
  • ✅ DNS, DHCP, TFTP, HTTP work together seamlessly
  • ✅ PXE boot works out-of-the-box for PowerVM
  • ✅ HAProxy provides free, integrated load balancing
  • ✅ All services configured for OpenShift requirements
• Isolation & Security:
  • ✅ Each cluster's DNS is isolated
  • ✅ Cluster compromise doesn't affect corporate DNS
  • ✅ Different security policies per cluster
  • ✅ Clear audit trail per cluster
• Hybrid Integration:
  • ✅ Bastion DNS forwards external queries to corporate DNS
  • ✅ Corporate DNS can delegate subdomains to bastion
  • ✅ Best of both worlds: self-service + corporate integration

🔄 DNS Delegation Strategy (Recommended)

Best Practice: Use DNS delegation to integrate bastion DNS with corporate DNS:

1. Corporate DNS delegates subdomain:
   • Corporate DNS: "For prod-power.company.com, ask bastion-1 at 10.1.1.10"
   • Corporate DNS: "For dev-power.company.com, ask bastion-2 at 10.1.1.20"
2. Bastion DNS is authoritative for its subdomain:
   • Bastion-1 answers: api.prod-power.company.com → 10.1.1.11
   • Bastion-1 answers: *.apps.prod-power.company.com → 10.1.1.11
3. Bastion DNS forwards other queries:
   • Query for google.com → Forward to corporate DNS
   • Query for other-cluster.company.com → Forward to corporate DNS

Result: Users can resolve cluster DNS from anywhere in the corporate network, but cluster teams have full control over their DNS zone.

📊 Comparison Table

🎯 Recommendation

Use Bastion DNS for Power clusters while keeping x86 clusters on corporate DNS if that's already working well.

• ✅ x86 clusters: Continue using corporate DNS if established processes work
• ✅ Power clusters: Use bastion DNS for speed, flexibility, and PowerVM-specific requirements
• ✅ Integration: Use DNS delegation so both approaches work together seamlessly
• ✅ Best of both: Corporate governance + self-service agility

📋 Installation Phases Explained

• Phase 1 - Bastion Setup (Steps 1-3): Provision and configure bastion LPAR with all required services (DNS, DHCP, TFTP, HTTP, HAProxy)
• Phase 2 - Create Discovery ISO (Steps 4-9): Use Red Hat Console to generate cluster-specific discovery ISO with embedded ignition configuration
• Phase 3 - Extract & Deploy (Steps 10-14): Download ISO, extract components, and deploy to bastion's PXE/HTTP directories
• Phase 4 - Network Boot (Steps 15-23): PXE boot all nodes, download RHCOS and ignition, register with Assisted Service
• Phase 5 - Monitor & Install (Steps 24-34): Monitor node discovery, configure cluster settings, start automated installation
• Phase 6 - Completion (Steps 35-40): Download credentials, verify cluster health

🔑 Key Components & Their Roles

• Bastion LPAR:
  • Runs dnsmasq (DNS, DHCP, TFTP for PXE boot)
  • Runs httpd (serves ignition files and RHCOS images)
  • Runs HAProxy (load balances API and ingress traffic)
  • Stores extracted ISO components
• Red Hat Console (Assisted Installer):
  • Generates cluster-specific discovery ISO
  • Manages cluster configuration and validation
  • Orchestrates installation process
  • Monitors installation progress and events
• Discovery ISO:
  • Contains RHCOS kernel, initramfs, and rootfs
  • Embeds cluster-specific ignition configuration
  • Includes Assisted Installer agent
  • Registers nodes with Assisted Service
• HMC/PowerVM:
  • Manages LPAR lifecycle
  • Executes lpar_netboot for PXE boot
  • Provides hardware virtualization

⚡ Automation with Ansible

The repository provides Ansible playbooks to automate most of these steps:

• setup-bastion.yaml: Automates Phase 1 (bastion setup)
• step-1-setup-services.yaml: Configures dnsmasq, httpd, HAProxy
• step-2-create-ignition.yaml: Downloads and extracts discovery ISO
• step-3-netboot-nodes.yaml: Executes lpar_netboot commands
• step-4-monitor.yaml: Monitors installation progress

Usage: ansible-playbook -e @vars.yaml playbooks/main.yaml

🆚 Assisted Installer vs Agent-Based Installer