Building a Private Cloud with OpenStack: From Bare Metal to Self-Service IaaS

Quick Summary (TL;DR) This handbook provides a complete blueprint for architecting, deploying, and operating a production-grade private cloud using OpenStack. You’ll learn how to select hardware, design resilient networks (VXLAN/DVR), choose storage backends (Ceph vs. LVM), automate deployment with Kolla-Ansible, and secure the platform with Keystone RBAC. We cover real-world edge cases including high-availability clustering, performance tuning, and common troubleshooting pitfalls. By the end, you’ll understand how to transform bare metal into a self-service IaaS platform that rivals public cloud providers in agility and control.

---

Introduction: Why OpenStack for Private Cloud?

The cloud computing landscape is dominated by hyperscalers—AWS, Azure, and GCP—offering unmatched scale and convenience. Yet, a growing cohort of enterprises is retreating from the public cloud, driven by data sovereignty laws, cost predictability concerns, and the need for low-latency, on-premises workloads. This is where OpenStack shines.

OpenStack is not a single product but an open-source cloud operating system that controls large pools of compute, storage, and networking resources throughout a datacenter. It provides a self-service dashboard (Horizon), an API layer, and CLI tools that mimic the public cloud experience—but on your hardware, under your control.

Why choose OpenStack over VMware or Proxmox? The answer lies in API compatibility (an emerging standard for hybrid cloud), telco-grade networking (via Neutron), and a vibrant ecosystem of deployment tools. While the learning curve is steep, the payoff is a platform that scales from a 3-node lab to thousands of nodes in a massive scale-out environment.

This handbook will walk you through every layer of the architecture, from physical hardware to tenant self-service, with a focus on production realities.

---

Architecture Blueprint: Core Components and Design Principles

Before touching a server, you must internalize OpenStack’s modular architecture. The project consists of dozens of services; the “Big Six” for a functional IaaS are:

Service	Code Name	Function
Compute	Nova	Manages the lifecycle of virtual machines (VMs)
Networking	Neutron	Provides network connectivity as a service (VPCs, routers, LBaaS)
Identity	Keystone	Centralized authentication and authorization
Image	Glance	Stores and retrieves virtual machine disk images
Block Storage	Cinder	Provides persistent block storage volumes to VMs
Dashboard	Horizon	Web-based self-service portal for users

Design Principle #1: Stateless Services, Stateful Backends The API services (Nova-API, Neutron-API, etc.) should be stateless. This means they can be scaled horizontally behind a load balancer without worrying about session affinity. All state—database records, message queue messages, and object storage—must live in dedicated backend clusters (MariaDB/Galera, RabbitMQ, Ceph).

Design Principle #2: Isolated Control and Data Planes Your physical network must be segmented. The control plane (API traffic, DB replication) should be isolated on a dedicated VLAN. The data plane (tenant VMs, storage replication) requires separate high-bandwidth links (10GbE or better). Never mix these on the same untagged interface.

Design Principle #3: The Cloud Controller Pattern In a highly available (HA) deployment, you’ll have a minimum of 3 controller nodes running all control services. These are backed by a clustered database (MariaDB Galera) and a mirrored message queue (RabbitMQ). Compute nodes are the workhorses, running the hypervisor (KVM) and networking agents.

graph TD
    subgraph "Control Plane (HA Cluster)"
        A[HAProxy Load Balancer] --> B[Controller Node 1]
        A --> C[Controller Node 2]
        A --> D[Controller Node 3]
        B --> E[(MariaDB Galera)]
        C --> E
        D --> E
        B --> F[RabbitMQ Cluster]
        C --> F
        D --> F
    end

    subgraph "Data Plane"
        G[Nova Compute Node 1] --> H[Neutron Agent]
        G --> I[Hypervisor KVM]
        J[Nova Compute Node 2] --> K[Neutron Agent]
        J --> L[Hypervisor KVM]
    end

    subgraph "Storage Backend"
        M[Ceph OSD Node 1]
        N[Ceph OSD Node 2]
        O[Ceph OSD Node 3]
    end

    B -- "Management API" --> G
    C -- "Management API" --> J
    G -- "vxlan data" --> H
    J -- "vxlan data" --> K
    G -- "RBD (Ceph)" --> M
    J -- "RBD (Ceph)" --> N
    E -- "Replication" --> E

---

Hardware Sizing and Bare Metal Provisioning

The biggest mistake in OpenStack projects is under-provisioning the control plane. Let’s set a baseline.

Reference Architecture for 100-200 VMs:

Node Type	Quantity	CPU	RAM	Disk (OS)	Disk (Data)	Network
Controller	3	16 cores	64 GB	2x 480GB SSD (RAID1)	N/A	2x 10GbE
Compute	5-10	32 cores	256 GB	2x 480GB SSD (RAID1)	N/A	2x 25GbE
Storage (Ceph)	3	8 cores	32 GB	2x 480GB SSD	10x 4TB NVMe	2x 25GbE

Bare Metal Provisioning Strategy: Do not manually install the OS on these nodes. Use PXE boot with a tool like MAAS (Metal-as-a-Service) or Ironic (OpenStack’s own bare metal service).

1. DHCP/DNS Setup: Configure the PXE boot environment to serve the Ubuntu Server 22.04 LTS image.
2. RAID Configuration: Use a BIOS/UEFI script to configure RAID arrays before the OS installer runs.
3. Automated Install: Use a preseed file or cloud-init to automate the OS installation, including SSH key injection and network bonding configuration.

Critical: Network Bonding. Always bond your physical NICs (LACP 802.3ad) for redundancy. The OpenStack services will not tolerate a flapping physical interface.

---

Network Design: From Flat to VXLAN and DVR

The network is the heart of OpenStack, and it’s where most projects fail. Neutron allows you to create complex virtual networks, but the underlying physical design must be flawless.

Layer 2 vs. Layer 3 (VXLAN) In small labs, a “flat” network (VLANs on a single switch) is acceptable. In production, you need VXLAN (Virtual Extensible LAN). VXLAN encapsulates Layer 2 frames in UDP packets (port 4789), allowing you to create millions of isolated networks across your existing Layer 3 fabric.

The MTU Trap: VXLAN adds 50 bytes of overhead. If your physical network MTU is 1500, your VMs will have an MTU of 1450. You must configure jumbo frames (MTU 9000) on the physical switches and hosts to avoid packet fragmentation and severe performance degradation.

DVR (Distributed Virtual Routing) In a traditional centralized routing model, all east-west traffic (VM-to-VM on different networks) must hairpin through a network node, creating a bottleneck. DVR solves this by placing routing logic on every compute node. This is mandatory for performance.

Configuration Snippet (ML2/OVS):

# /etc/neutron/plugins/ml2/ml2_conf.ini
[ml2]
type_drivers = flat,vlan,vxlan
tenant_network_types = vxlan
mechanism_drivers = openvswitch,l2population

[ml2_type_vxlan]
vni_ranges = 1:1000
vxlan_group = 239.1.1.1  # Multicast group for discovery (or use L2 population)

flowchart LR
    A[Tenant VM 1] -->|"VXLAN (UDP 4789)"| B[Compute Node OVS]
    C[Tenant VM 2] -->|"VXLAN (UDP 4789)"| D[Compute Node OVS]
    B -->|"IP Fabric (L3)"| E[Spine Switch]
    D -->|"IP Fabric (L3)"| E
    E --> F[External Router]
    F --> G[Internet/Corporate Net]
    style B fill:#f9f,stroke:#333,stroke-width:2px
    style D fill:#f9f,stroke:#333,stroke-width:2px

---

Storage Backends: Ceph, LVM, and NFS

Storage is a religious war in OpenStack. You have three primary choices for Cinder (block storage) and Glance (images).

1. LVM (Logical Volume Manager)

• Pros: Simple, low overhead, no extra services.
• Cons: Single point of failure (unless using DRBD), no replication, limited to a single host.
• Use Case: Dev/test environments, edge sites with local storage requirements.

2. NFS (Network File System)

• Pros: Easy to set up, familiar.
• Cons: Poor performance for block workloads, single point of failure (unless using a clustered NFS like GlusterFS).
• Use Case: Storing images for Glance, not for running VMs.

3. Ceph (The Production Standard) Ceph provides RBD (RADOS Block Device) which is the de-facto standard for OpenStack storage.

• Pros: Self-healing, distributed, no single point of failure. Provides block (Cinder), object (Swift/S3), and file (CephFS) storage.
• Cons: Resource-hungry (needs dedicated nodes), network-intensive, complex to tune.

Ceph Best Practices:

• Separate Clusters: Do not run Ceph OSDs on compute nodes if you can avoid it. The I/O contention will hurt both workloads.
• Network Separation: Ceph public network (client-facing) and cluster network (replication) must be on different physical NICs.
• CRUSH Map: Define failure domains (host, rack, row) to ensure replicas are spread across physical boundaries.

Cinder Volume Creation Flow:

1. User requests a 100GB volume.
2. Cinder API validates the request via Keystone.
3. Cinder Scheduler picks a storage backend (e.g., Ceph).
4. Cinder Volume Manager creates the RBD image.
5. Nova attaches the volume to the VM as a virtio-scsi device.

---

Deployment Automation: Kolla-Ansible, TripleO, and Charms

You cannot manually install OpenStack. It’s too complex. You need an orchestration tool. Here’s the landscape:

Tool	Package Type	Strengths	Weaknesses	Best For
Kolla-Ansible	Docker Containers	Simplicity, upgradeability, clean lifecycle	Requires Docker expertise	Most users (Recommended)
TripleO	RPMs via Heat	Native RHEL/CentOS support, deep integration	Complex, steep learning curve	RHEL-centric large enterprises
OpenStack Charms	Snaps via Juju	Flexible, model-driven, supports hybrid (bare metal + containers)	Juju curve, smaller community	Canonical-focused environments

Why Kolla-Ansible is the winner: Kolla builds Docker images for every OpenStack service. Ansible handles the orchestration—pulling images, generating configs, and starting containers.

Deployment Steps (Kolla-Ansible):

1. Prepare the Deployment Host: Install Ansible, Docker, and Kolla-Ansible.
2. Inventory: Define your hosts in inventory/all-in-one or a custom multi-node inventory.
3. Global Config: Edit /etc/kolla/globals.yml:

   kolla_base_distro: "ubuntu"
   kolla_install_type: "source"  # or binary
   network_interface: "ens192"   # Management interface
   neutron_external_interface: "ens224" # External interface for floating IPs
   enable_haproxy: "yes"
   enable_cinder: "yes"
   enable_cinder_backend_lvm: "no"
   enable_ceph: "yes"
   

4. Bootstrap: kolla-ansible bootstrap-servers (installs Docker on all nodes).
5. Pre-check: kolla-ansible prechecks (validates hardware and network).
6. Deploy: kolla-ansible deploy (runs the full deployment).
7. Post-Deploy: kolla-ansible post-deploy (generates the /etc/kolla/admin-openrc.sh file).

The upgrade path: kolla-ansible upgrade handles rolling upgrades of containers, which is a massive advantage over RPM-based systems that often require downtime.

---

Identity and Security: Keystone, RBAC, and TLS

Security is not an afterthought; it’s the foundation. Keystone is the gatekeeper.

Keystone Concepts:

• Projects (Tenants): Isolated containers for resources. Think of them as AWS Accounts.
• Users: Identity for humans or services.
• Roles: Define permissions (e.g., admin, member, reader).
• Domains: High-level containers for projects and users (useful for multi-tenant enterprises).

RBAC in Practice: You must move beyond the default admin and member roles. Create custom roles for granular control:

# Create a role for network admins
openstack role create network_admin

# Assign this role to a user in a specific project
openstack role add --user john --project prod_cloud --role network_admin

Hardening Checklist:

1. TLS Everywhere: Terminate TLS at HAProxy. Use a trusted CA (Let’s Encrypt or internal PKI). Never expose APIs on plain HTTP.
2. Fernet Tokens: Ensure Keystone is using Fernet tokens (default in Kolla). They are stateless and cryptographically signed.
3. Service Isolation: Each OpenStack service should have its own service account with minimal permissions.
4. Domain Config: Use LDAP/AD integration for user management via Keystone Federation (SAML) or LDAP backend.

Network Security:

• Security Groups: These are stateful firewalls. Ensure default rules deny all ingress.
• RBAC on Networks: Use Neutron RBAC to allow specific projects to share networks without giving them global admin.

---

Self-Service IaaS: Enabling Tenants with Horizon and CLI

The goal is to make your users independent of the IT team. They should be able to spin up resources without a ticket.

The User Experience Flow:

1. Login to Horizon: User authenticates against Keystone.
2. Create Network: User defines a private network (10.0.0.0/24).
3. Boot Instance: User selects an image (Ubuntu 22.04), chooses a flavor (m1.small), attaches a security group, and clicks “Launch”.
4. Assign Floating IP: User requests a public IP from the external pool and associates it with the VM.

Enabling this requires careful configuration:

Flavors: Define standard flavors (m1.small = 1 vCPU/2GB, m1.medium = 2 vCPU/4GB, etc.). Quotas are essential to prevent resource exhaustion.

# Set quotas for a project
openstack quota set --instances 50 --cores 100 --ram 256000 --volumes 50 --gigabytes 1000 prod_cloud

Horizon Customization: For a branded experience, you can customize Horizon’s logo and dashboard. More importantly, you can disable panels that you don’t want users to see (e.g., admin panels for non-admin users).

CLI Automation: Power users will want the CLI. Provide a clouds.yaml file to simplify configuration:

clouds:
  mycloud:
    auth:
      auth_url: https://cloud.example.com:5000/v3
      project_name: prod_cloud
      username: user
      password: pass
    region_name: RegionOne

---

Production Edge Cases: High Availability, Scaling, and Troubleshooting

This is where the rubber meets the road. Your deployment will fail. Here’s how to survive.

High Availability (HA) Architecture:

• Database: MariaDB Galera Cluster (3 nodes). Synchronous replication.
• Message Queue: RabbitMQ in mirrored queue mode (quorum queues in newer versions).
• API Layer: HAProxy on the controllers, load balancing to all three API nodes.
• Storage: Ceph handles its own HA; ensure min_size=2 so writes block if one OSD is down.

Scaling Compute: When you run out of compute capacity, add nodes:

1. Install the OS and Docker.
2. Add the node to the Ansible inventory.
3. Run kolla-ansible deploy --limit new-node.
4. The node will automatically join the cluster and start receiving VMs.

Troubleshooting Toolkit:

• Check Container Status: docker ps on the relevant node.
• View Logs: docker logs <container_name> (e.g., docker logs nova_compute).
• Message Queue Issues: If VMs are stuck in “Build”, check RabbitMQ: rabbitmqctl list_queues.
• Database Locking: If Galera is down, check SHOW STATUS LIKE 'wsrep%';.

Common Edge Case: The “Split-Brain” Scenario. If the network between controllers fails, Galera may partition. To avoid this, use Galera’s arbitrator (garbd). It acts as a tie-breaker, ensuring only one partition remains primary.

Performance Degradation: If CPU steal time is high on compute nodes, check for noisy neighbors. Use CPU pinning for critical workloads:

# Define a flavor with dedicated CPU policy
openstack flavor create --vcpus 4 --ram 8192 --disk 80 m1.cpu_pinned
openstack flavor set --property hw:cpu_policy=dedicated m1.cpu_pinned

---

Performance Benchmarks and Optimization

Don’t guess; measure. Here are standard benchmarks and expected outcomes.

Benchmarking Tools:

• CPU/Memory: sysbench, unixbench.
• Storage: fio (inside a VM and on the Ceph cluster).
• Network: iperf3 (east-west traffic), qperf (latency).

Baseline Expectations (Ceph-backed, 25GbE network):

Metric	Value (Target)
VM Disk Sequential Read (4K)	> 50,000 IOPS
VM Disk Sequential Write (4K)	> 30,000 IOPS
East-West Network Throughput	> 8 Gbps (due to VXLAN overhead)
API Response Time (Nova)	< 200 ms
Instance Boot Time (from cache)	< 30 seconds

Optimization Levers:

1. Ceph Tuning: Ensure rbd cache is enabled on compute nodes. Set bluestore_compression_mode to none for better performance unless you need space savings.
2. NUMA Topology: For memory-intensive workloads, define NUMA-aware flavors to prevent cross-socket memory access.
3. OVS Tuning: Enable userspace_tso and dpdk for high-throughput workloads (though this adds complexity).
4. Glance Cache: Pre-cache popular images on compute nodes to speed up instance spawns.

---

Conclusion: From Bare Metal to Self-Service IaaS

Building a private cloud with OpenStack is a journey, not a destination. It requires a shift in mindset from “managing servers” to “providing a platform.” The architecture we’ve covered—modular controllers, VXLAN networking with DVR, Ceph-backed storage, and Kolla-Ansible automation—provides a blueprint for a system that can scale to hundreds of nodes while offering users the self-service agility of a public cloud.

The path is challenging, but the rewards are substantial: data sovereignty, cost control, and absolute ownership of your infrastructure. By focusing on the core principles of stateless services, isolated networks, and robust automation, you can transform your bare metal into a cloud that your developers will love and your CFO will approve.

Start small. Build a proof of concept. Break it. Fix it. And then scale.

---

FAQ

Q: What are the minimum hardware requirements for a small OpenStack deployment? A: For a minimal production-like environment, you need at least 3 nodes: 1 controller (8 vCPUs, 16GB RAM, 100GB disk) and 2 compute nodes (each with 8 vCPUs, 32GB RAM, and local storage). For storage, consider a dedicated Ceph cluster with at least 3 nodes. However, for a homelab, you can start with a single node using virtualization, but it’s not recommended for production.

Q: How do I choose between Kolla-Ansible, TripleO, and Charms for deployment? A: Kolla-Ansible is the most popular for its simplicity and containerized services, making it ideal for most deployments. TripleO uses Heat and is more complex, suited for large-scale RHEL-based environments. Charms are Juju-based, offering flexible orchestration, but have a steeper learning curve. For most users, Kolla-Ansible is recommended.

Q: What are the best practices for securing OpenStack APIs? A: Always terminate TLS at the load balancer or proxy, use Keystone for authentication with Fernet tokens, enable RBAC with domain-specific roles, and isolate the internal API network. Additionally, use HAProxy for high availability and set up SSL certificates from a trusted CA.

Q: How can I ensure high availability for the control plane? A: Deploy at least 3 controller nodes behind a load balancer (HAProxy) for all API services. Use MariaDB Galera for database clustering, RabbitMQ in mirrored queues, and Memcached with multiple nodes. Also, use Ceph for storage to avoid single points of failure.

Q: What are the common networking pitfalls when setting up OpenStack? A: Common pitfalls include misconfigured MTU sizes (must be 1500 or lower for VXLAN), incorrect VLAN trunking on physical switches, and not enabling IP forwarding on network nodes. Also, ensure that the DHCP agent is properly configured and that the metadata service is reachable from instances.