On one hand, OpenStack is quite promising with open APIs, open-source reference implementation with extensible plug-in architecture to support different implementations. On the other hand, it is a nightmare for software developers building enterprise-grade distributed applications on-top-of OpenStack i.e., using only APIs available to users/tenants. Such applications need to be elastic (scale-out and scale-in as loads fluctuate), highly-available (several 9's of availability), and support high throughput (several Gbps of traffic). Unfortunately, many of the primitives available in the physical infrastructure to be able to build such services are either non-existent or available via ad-hoc extension APIs at the virtual infrastructure layer in OpenStack implementations.
The Good (examples):
- Nova's CRUD API for VMs: The basic VM creation and deletion with several options are pretty solid APIs in the OpenStack and this allows an application orchestration engine to easily scale-out and scale-in the app VMs as needed. The placement options exist albeit bit restricted.
- Neutron's CRUD API for networks, ports, router, subnets: Neutron abstractions of the networks, ports, routers, and subnets makes it easy for an application orchestration engine to connect VMs to right networks for creating multiple app tiers with proper isolation.
The Bad (examples):
- No notification APIs from OpenStack services such as Nova, Neutron, Glance, and Keystone: There is no way for the application orchestrator process to timely detect rebooted/deleted/dead application VMs or ports. The only option is to periodically check.
- Security groups API mess: Depending on whether Nova is implementing the security groups or Neutron, the semantics of the APIs are different!!
- Network performance: The reference implementation's stack is too complex (layers of different bridges connected by veth pairs, iptables with resource intensive nf_conntrak) with multiple bottleneck locations. And different vendors' plugins have myriad of other limitations.
The Ugly (examples):
- Too much restriction on network connectivity without exposing proper APIs to customize: For example, source IP address spoofing is not allowed even in a local network, where as that is an essential primitive for building highly available services. Ad-hoc neutron extension APIs such as allowed-address-pairs and port-security APIs alleviate this but not being part of core APIs means it is not no guarantee that they are available everywhere
- Semantics of APIs left to the interpretation of plugins: for example, in a physical world, all servers connected to a switch can communicate at L2 without any restriction. However, with all of the neutron plugins, there is one or another restriction imposed that is not explicitly indicated (for example, can't have the multiple fixed IPs on a port, can't have the same IP on two ports, etc.)
