Last year, we established a dedicated lab, an isolated environment distinct from production, situated on separate hardware with its own domain and vCenter. It serves as a playground for employees to experiment, test, and innovate without affecting the production environment.
While I understand your apprehensions about potential disruptions when everyone is free to experiment and potentially disrupt things, we’ve taken steps to address such concerns. To mitigate risks, we implemented the SDDC.lab project, a collaborative initiative developed by Rutger Blom and Luis Chanu, utilizing Ansible. This project enables us to create nested pods, each comprising essential components:
- VyOS router for north-south communication
- ESXi hosts, organized into two clusters – one for workloads and another for NSX Edge nodes
- vCenter server
- NSX Manager
- NSX Edge nodes
- A jump host facilitating access to the pod
Furthermore, we have the flexibility to customize configurations by optionally incorporating the following components:
- VMware Aria Operations for Logs (vRealize Log Insight)
- NSX Global Manager
- NSX Advanced Load Balancer (Avi)
- VMware Aria Operations for Networks (vRealize Network Insight)
- Management cluster featuring three ESXi hosts
- Tanzu
- Workload VMs
Importantly, all these pods exist within a segregated environment, forming a distinct bubble. This isolation allows individuals to experiment freely within their designated pods without impacting other pods or the broader production system.
Use Cases:
Here are some particular instances in which I recently utilized my pod: I employed it for testing and scripting my custom attributes script, as well as for evaluating firewall policies in NSX. Additionally, we faced the task of migrating production DFS to a new server. My colleague informed me about a server in a segmented zone using a DFS path, necessitating a downtime-free migration – a scenario ideal for testing within the Lab.
For this, I set up essential servers and configured the necessary components, including an AD domain, DFS, and a file server, essentially creating a replica of the production environment. Rather than utilizing a physical firewall, I implemented policies within NSX.
Beyond the firewall policy, I had to define services, referring to the Microsoft “Service overview and network port requirements.” While some ports were specified in the AD services, the only service missing was SMB. I crafted a policy, specifying the source and destination addresses.
We experimented with the to-be configuration, migrating DFS services to the new server and adjusting the destination address in the policy until arriving at a functional new configuration.
Armed with this information, we generated a change request for the network team. On migration day, the process involved reconfiguring DFS and exercising patience. Fortunately, we executed the migration seamlessly without downtime or impact on production.
In conclusion, I advocate for every systems team to have a dedicated lab or test environment. Considering that development teams commonly have separate environments for development, testing, UAT, and production servers, my suggestion is not necessarily for everyone to adopt nested pods, but at least to maintain a distinct environment for testing and experimentation.
