🔧 Fix: Live Environment Out of Space
If the SNO node is running from PXE/live environment and shows root filesystem at 100%, you need to remount with more space before coreos-installer can run.
Quick Fix
# On SNO node - remount /run with more space
sudo mount -o remount,size=8G /run
# Verify
df -h /run
# Now install RHCOS to disk
sudo coreos-installer install /dev/sda \
--ignition-url http://10.0.10.191:8000/ignition/master-sno.ign \
--insecure-ignition
# After installation completes, reboot
sudo reboot
Note: Use
--insecure-ignition when fetching ignition over HTTP. For HTTPS, you can omit this flag or use --ignition-hash instead.
📊 Track Bootstrap Progress
On SNO Node
# SSH to SNO node
ssh core@10.0.10.103
# Watch bootkube service (main bootstrap service)
sudo journalctl -u bootkube.service -f
# Check bootkube status
sudo systemctl status bootkube.service
# Watch containers starting
watch -n 5 'sudo crictl ps | wc -l'
# List all running containers
sudo crictl ps
# Check kubelet status
sudo systemctl status kubelet
# Follow kubelet logs
sudo journalctl -u kubelet -f
From Bastion (Recommended)
# Set kubeconfig
export KUBECONFIG=~/workdir-ocp4-ai/auth/kubeconfig
# Watch cluster operators
watch -n 10 'oc get co'
# Watch nodes
watch -n 10 'oc get nodes'
# Use openshift-install to monitor
cd ~/workdir-ocp4-ai
openshift-install wait-for install-complete --log-level=info
⏱️ Bootstrap Timeline
0-5 minutes: Node boots, kubelet starts, bootkube service begins
5-15 minutes: Static pods start (etcd, API server, controller manager)
15-25 minutes: API server becomes available, cluster operators start deploying
25-40 minutes: All cluster operators become available
40-60 minutes: Installation complete ✅
✅ Quick Status Check
One-Liner Status Check
# From bastion
export KUBECONFIG=~/workdir-ocp4-ai/auth/kubeconfig
oc get clusterversion && oc get nodes && oc get co
Status Check Script
#!/bin/bash
# Save as check-sno-status.sh on bastion
export KUBECONFIG=~/workdir-ocp4-ai/auth/kubeconfig
echo "=== SNO Installation Status ==="
echo ""
echo "Node Status:"
oc get nodes 2>/dev/null || echo " API not ready yet"
echo ""
echo "Cluster Operators:"
oc get co 2>/dev/null | head -10 || echo " API not ready yet"
echo ""
echo "Cluster Version:"
oc get clusterversion 2>/dev/null || echo " API not ready yet"
# Make it executable and run
chmod +x check-sno-status.sh
watch -n 30 ./check-sno-status.sh
🎯 Key Milestones to Watch
On SNO Node
# 1. Check if bootkube started
sudo journalctl -u bootkube.service | grep "bootkube.service: Started"
# 2. Check if static pods are running
sudo crictl ps | grep -E "etcd|kube-apiserver|kube-controller"
# 3. Check if bootstrap completed
sudo journalctl -u bootkube.service | grep -i "bootstrap complete"
From Bastion
export KUBECONFIG=~/workdir-ocp4-ai/auth/kubeconfig
# 1. Check if API server is accessible
oc version
# 2. Check if node appears
oc get nodes
# 3. Check cluster operators progress
oc get co
# 4. Check if all operators are available
oc get co | grep -v "True.*False.*False"
🔍 Verify Installation Success
Installation is complete when you see:
- ✅
oc get nodesshows node as "Ready" - ✅
oc get coshows all operators "Available=True, Progressing=False, Degraded=False" - ✅
oc get clusterversionshows "AVAILABLE=True" - ✅ Console URL is accessible
- ✅ Can login with kubeadmin credentials
Access the Cluster
# Get console URL
oc whoami --show-console
# Get kubeadmin password
cat ~/workdir-ocp4-ai/auth/kubeadmin-password
# Login via CLI
oc login -u kubeadmin -p $(cat ~/workdir-ocp4-ai/auth/kubeadmin-password)
⚠️ Common Issues & Quick Fixes
Issue: API Server Not Responding
# Check if API server pod is running
ssh core@10.0.10.103
sudo crictl ps | grep kube-apiserver
# Check API server logs
sudo crictl logs $(sudo crictl ps | grep kube-apiserver | awk '{print $1}')
Issue: Node Not Ready
# Check node conditions
oc describe node | grep -A 10 Conditions
# Check kubelet on node
ssh core@10.0.10.103
sudo systemctl status kubelet
sudo journalctl -u kubelet -n 50
Issue: Cluster Operators Degraded
# Find degraded operators
oc get co | grep -v "True.*False.*False"
# Get details on specific operator
oc describe co
# Check operator pods
oc get pods -n openshift-
💡 Useful Aliases
Add these to your ~/.bashrc on the bastion for quick access:
# Add to ~/.bashrc
export KUBECONFIG=~/workdir-ocp4-ai/auth/kubeconfig
# Quick aliases
alias sno-ssh='ssh core@10.0.10.103'
alias sno-status='oc get nodes && oc get co'
alias sno-watch='watch -n 10 "oc get nodes && echo && oc get co"'
alias sno-logs='ssh core@10.0.10.103 "sudo journalctl -u bootkube.service -f"'
alias sno-pods='oc get pods -A'
# Then reload
source ~/.bashrc