In this post I will detail how you can expand Cisco’s CML2 capabilities to support Cisco Viptela images for CCIE Enterprise Infrastructure labbing purposes.

Guide Version: 1.0
Last Updated: 25 September 2021
CML2 Version Tested: 2.0.0-b13
Viptela Firmware Versions Tested: 18.X, 19.X, 20.X
Github Documentation Link: [docx] [pdf]
Github Viptela Serial File Link: [link]

Important Disclaimer: This document is an unofficial resource that has been created to detail how Cisco Modeling Labs 2 (“CML2”) can be configured to support Cisco SD-WAN images. It is an educational resource that has been developed to support the wider CCIE community for use in a non-production environment. The author does not recommend implementing any custom functionality into a production deployment of CML2 without first consulting with an official Cisco representative, agent, affiliate or partner.

By continuing with this document, you automatically acknowledge that ether-net and its author(s) are not responsible for any damages that may arise within your technical environment which may result from the indirect or direct usage of this guide. You also acknowledge that no support or warranties are extended by ether-net should you utilise this document.

 

Table of Contents

1. Pre-Requisites
2. Obtaining Cisco SD-WAN Images
3. Preparing CML2 for Custom Images
4. Uploading Cisco SD-WAN Images to CML2
5. Creating Cisco SD-WAN Node Definitions
6. Creating the Cisco SD-WAN Image Definitions
7. Building the Test Lab
8. Recommended Node Start-Up Sequence
9. Next Steps & Additional Resources
10. Known Issues

 

Pre-Requisites

The following pre-requisites must be met before commencing with this guide:

• A host machine that meets the following minimum system requirements:
  • Intel CPU with at least 16 cores (recommended: 24+)
  • 64GB RAM (recommended: 128GB for larger topologies)
  • 100-150GB free disk space
• An installed and licensed copy of Cisco Modeling Labs 2 (“CML2”) on the above mentioned host machine
• Familiarity with the administration of CML2 and a hypervisor tool (e.g.: vmWare, KVM)
• A valid Cisco support contract that enables you to download SD-WAN QCOW2 images

IMPORTANT NOTE: Consult the Cisco SD-WAN Server Recommendations documentation [link] if you intend to use your CML2 Cisco SD-WAN lab environment as an official “test” environment within your business.

 

Obtaining Cisco SD-WAN Images

Section Purpose
To provide information on how Cisco SD-WAN images can be legally acquired.

Notes
This section can be skipped if you already possess Cisco SD-WAN images.

---

Step 1: In your web browser navigate to the “CCIE Enterprise Infrastructure Equipment and Software List” and verify what software versions are being utilised for the Cisco CSR 1000v and Cisco SD-WAN products [link].

This guide assumes that Cisco CSR1000v is using Cisco IOS XE SD-WAN Release 16.12 and that Cisco SD-WAN products are using software version 18.4.

Step 2: In your web browser navigate to the Cisco Software Center page for SD-WAN images [link].

Step 3: For each of the vEDGE Cloud, vManage Software and vSmart Software perform the following actions:

• Click on the hyperlink to the relevant software (e.g.: vEDGE Cloud)
• Use the left-hand sidebar to browse to the desired software version and then click on it
• Click the download button for the QCOW2 image, read and agree to the terms of service for software download, and then download the image
• If you wish to practice software upgrades through vManage then additionally download a release version that is one down from your current one (e.g.: 18.3.8 where 18.4.5 is the image that you are labbing on).
• It is recommended to download the vEDGE Cloud twice and then rename the second file to “viptela-vbond-X.X.X-genericx86-64” so that you can maintain a different image set for vBond use.

 

If you cannot download the image due to the following prompt then either log in with your Cisco account OR contact your account administrator / Cisco account manager to identify what steps need to be taken to download the images.

 

Preparing CML2 for Custom Images

Section Purpose
To ensure that enough resources have been allocated to the CML2 virtual machine so that it can support both the storage and mounting of Cisco SD-WAN images.

Notes
This section can be skipped if your CML2 virtual machine has a 100-150 GB virtual disk attached to it.

---

Step 1: Gracefully shutdown your CML2 virtual machine – DO NOT PROCEED UNTIL THIS HAS BEEN DONE!

Step 2: In your hypervisor application expand the virtual disk allocated to CML2 to be in the range of 100-150GB (smaller labs):

• VMware ESXI / Hypervisor / vSphere: [Official KB Link]
• VMware Fusion: [Official KB Link]
• VMware Workstation: [Official KB Link]

Step 2: Example shown with VMware Fusion 11.5.6 on macOS

Step 3: Power on the CML2 virtual machine and ensure that no labs are running.

Step 4: In your web browser navigate to the CML2 CentOS Linux web interface by entering: https://<CML2>:9090 ; where <CML2> is the IP address or hostname that you configured during initial system setup. Log in with an account that possesses administrative privileges and check the box that states “Reuse my password for privileged tasks”.

 

 

Step 5: In the left-hand sidebar click on the “Storage” link and then click on the “/dev/cl/root” file system whose mount point is “/”. This will take you to a new page.

 

Step 6: On the right-hand pane a box called “Physical Volumes” is present. Click the “+” button and note that an “Add Disks” window appears. Within the “Add Disks” window click the “Add” button to allocate the additional space provisioned in Step 2 to your CML2 root file system.

 

Step 7: Note that the disk we added in Step 6 now appears in the “Physical Volumes” section of the page. Now click on the “Grow” button. This will launch a new window called “Grow Logical Volume” – slide the box all the way to the right and then click “Grow” to ensure that the “/dev/cl/root” file system is fully expanded.

 

Step 8: In the left-hand sidebar click on the “Storage” link and verify that the CML2 root file system has grown. NOTE: A small amount of allocated disk space will be lost due to file system overhead. Once you have completed this step sign out of the CML2 CentOS Linux web interface.

Step 9: In your web browser navigate to the CML2 Lab Manager page. Login with an account that has administrative privileges and note that the “Disk [GB]” will display the same amount of total disk space observed in the previous step.

 

Uploading Cisco SD-WAN Images to CML2

Section Purpose
To advise how to upload and link Cisco SD-WAN images to the node definitions that were created

Section Pre-Requisites

• Completed all steps in “Obtaining Cisco SD-WAN Images” OR you have access to vManage/vSmart/vEdge Cisco SD-WAN images; AND
• Completed all steps in “Preparing CML2 for Custom Images”.

---

Step 1: In your web browser navigate to your CML2 Lab Manager instance and log in with an administrator account. Click on Tools > Node and Image Definitions.

Step 2: At the top of the page click the “Manage Image Definitions” button.

Step 3: Perform the following steps for each vManage, vSmart, vBond (if optionally created from vEdge image) and vEdge image you wish to upload:

• Click the “Browse” button
• Select one QCOW2 image that you wish to upload
• Click the “Upload” button and wait for the image to upload – a progress bar will indicate how long it will take for the image to complete uploading. Note that after the progress bar hits 100% it may take some time for the uploaded file to be processed – be patient!
• Verify that the image appears in the “Uploaded Images” section

 

IMPORTANT NOTE: You may experience an error where the API times out while uploading images (Error 500: API Not Ready). If this occurs wait 3-5 minutes and then refresh the page as the image may have uploaded but could have taken time to be processed. If this does not happen then try to re-upload the image. Where you experience multiple upload failures you can use SCP to copy images to the server with the command: “scp <FILE_NAME> admin@<CML2_IP>:”.

 

Creating Cisco SD-WAN Node Definitions

Section Purpose
To advise what custom node definitions are required to support Viptela image operations

Note
Sample CML2 node definition YAML files have been provided to download and upload to your CML2 instance on the ether-net’s GitHub [GitHub link]

---

Step 1: In your web browser navigate to your CML2 Lab Manager instance and log in with an administrator account. Click on Tools > Node and Image Definitions.

 

Step 2: Scroll down the page until you see the Nodes heading and then click the “Create New Node Definition” button.

 

Step 3: Create a new node definition for the vManage, vBond, vSmart and vEdge devices with the below  properties. Where a specific property is not listed it can be of any value and not impact the Viptela device from operating (e.g.: General > Description can be anything as it’s just metadata, so it won’t be included below).

 

Critical Information

• Creating a separate node or image definition for vBond and vEdge is not required but can make the maintenance of automated lab builds easier, particularly if you wish to maintain individual images for these devices.
• The nature of ALL devices MUST be router while they MUST be simulated as servers – if you change these settings you will not be able to spin the node up!
• DO NOT specify a boot drive for any of the nodes otherwise the image WILL NOT be mounted to the node when you try to start it in a lab.
• It is important that you copy the interface mapping EXACTLY otherwise you WILL experience interface mismatch issues (e.g.: CML2 will claim you are attaching to eth0 on the node when in actuality you’ll map to a random interface that may or may not exist!).
• Monitor the vManage database capacity on your running node as you expand your SD-WAN lab so that you can optimize the data disk size.

Generic Viptela Node Properties

All Viptela nodes regardless of their type must possess these properties

General

• Nature: router

Linux Native Simulation

• Domain Driver: KVM
• Simulation Driver: server
• Network Driver: VirtIO

pyAts

• Enable pyATS: False

Configuration

• Configuration Generator: server

Provisioning

• Enable provisioning: False

 

vManage Specific Node Properties

User Interface

• Visible: true
• Prefix: vManage-
• Icon: server
• Label: Viptela vManage

Linux Native Simulation

• Disk Driver: IDE (if you do not specify IDE your vManage image will become unstable)
• Memory: 16000  (recommended: 24000 to 32000)
• CPUs: 6  (increment by 2 for each site you add with 2x wEdges after exceeding 5x sites and 10x wEdges)
• Data Disk Size: 30  (This is what vManage will use for its database NOT the boot disk! Set to 100GB for always on vManage with <= 250 sites being simulated)

Interfaces

• Has a Loopback Interface: true
• Loopback name: system
• Number of serial ports: 1
• Default number of physical interfaces: 2
• Interfaces:
  • eth0
  • eth1

Boot

• Timeout (seconds): 300

 

vSmart Node Properties

User Interface

• Visible: true
• Prefix: vSmart-
• Icon: router
• Label: Viptela vSmart

Linux Native Simulation

• Memory: 2048  (recommended: 4096 for routing intensive labs)
• CPUs: 2  (recommended: 4 for routing intensive labs)

Interfaces

• Has a Loopback Interface: true
• Loopback name: system
• Number of serial ports: 1
• Default number of physical interfaces: 2
• Interfaces:
  • eth0
  • eth1

Boot

• Timeout (seconds): 60

 

vBond & vEdge Node Properties

User Interface

• Visible: true
• Prefix: vEdge-  OR vBond-
• Icon: router
• Label: Viptela vSmart  OR Viptela vBond

Linux Native Simulation

• Memory: 2048  (recommended: 4096 for larger scale labs)
• CPUs: 2

Interfaces – vEdge Specific

• Has a Loopback Interface: false
• Number of serial ports: 1
• Default number of physical interfaces: 6
• Interfaces:
  • eth0
  • ge0/0
  • ge0/1
  • ge0/2
  • ge0/3
  • ge0/4

Interfaces – vBond Specific

• Has a Loopback Interface: false
• Number of serial ports: 1
• Default number of physical interfaces: 6
• Interfaces:
  • eth0
  • ge0/

Boot

• Timeout (seconds): 60

 

Creating the Cisco SD-WAN Image Definitions

Section Purpose
To advise how to upload and link Cisco SD-WAN images to the node definitions that were created

Section Pre-Requisites

• Completed all steps in “Uploading Cisco SD-WAN Images to CML2”; AND
• Completed all steps in “Creating Cisco SD-WAN Node Definitions”.

---

Step 1: In your web browser navigate to your CML2 Lab Manager instance and log in with an administrator account. Click on Tools > Node and Image Definitions.

 

Step 2: At the top of the page click the “Create New Image Definition” button

 

Step 3: For each vManage, vBond, vSmart and vEdge device complete the following steps:

1. ID: Enter a unique identifier to identify this specific image. This cannot be changed once created!
2. Label: Enter a label name for the image. This is what will show up as as the Image Definition when a node is selected in a lab workspace
3. Description: Enter a description for the image. I copy/paste the label into this field as it displays in in the Node & Image Definitions page list.
4. Disk Image:  Select the corresponding QCOW2 image that you uploaded. E.g.: If you are creating a vManage image definition then you will want to select the vManage QCOW2 image.
5. Node Definition: Select the corresponding node definition for the QCOW2 image. A node can have many images mapped to it but an image can only ever be mapped to a single node definition.
6. Ignore all Linux Native Simulation configuration fields unless you wish to custom allocate resources on an image-per-image basi.
7. Click the “Create Image Definition” button once completed to finalise the image creation process.

 

 

Building the Test Lab

Section Purpose
To verify that Cisco SD-WAN nodes boot correctly within CML2 Lab Manager

Section Pre-Requisites

• Completed all prior sections successfully.

---

Step 1: Download the “SDWAN Test Lab” YAML file from the ether-net’s GitHub and then open it in a text editor [link]

Step 2: Verify that the node_definition and image_definition values between lines 155 and 247 in the “SDWAN Test Lab” YAML file match the ID field for your node and image definitions. You may experience issues importing the lab or booting nodes if mismatches are present!

Step 3: In the CML2 Lab Manager workspace click the “Import Lab” button to be taken to the Import Lab Page.

Step 4: Click the “Browse” button and navigate to the directory where the “SDWAN Test Lab.yaml” file is stored so that you can select it.

 

Step 5: Click the “Upload Topology” button to upload the YAML file.

 

Step 6: Once the lab has finished importing click the “Go to Lab” button to go directly to the lab workspace.

Step 7: Start the nodes in the following order:

1. Desktop
2. VPN512-SW
3. vManage
4. vSmart
5. vBond
6. vEdge

Step 8: Once all nodes have started click on the “Lab Notes” tab and follow node configuration instructions.

Step 9: After completing all of the Lab Notes instructions you should be able to…:

• Successfully access vManage GUI from the Desktop’s Firefox web browser

• Successfully ping all VPN512 interface addresses from all nodes

 

Recommended Node Start-Up Sequence

Section Purpose: To provide advice on how you should start your Cisco SD-WAN lab to avoid arbitrary issues.

This node start-up sequence has arisen because it seemingly resolves odd issues with Cisco SD-WAN images. I cannot definitively prove this resolves issues that I have experienced but the odd issues seemingly go away when I follow this sequence.

The recommended node spin up sequence is:

1. Any host VM devices that are used to access vManage;
2. vManage nodes – do not start 1 vManage until you’ve managed to log into the prior one’s CLI and successfully logged into the web interface. If you are running clusters boot 1 cluster before attempting another.
3. vSmart nodes – sets up the control plane, if using Affinity Pairing do 1x pair at a time
4. vBond nodes – addresses random cEdge/vEdge fabric authentication issues
5. cEdge / vEdge nodes – do 1x site at a time
6. Site nodes residing in service VPNs (e.g.: IOSv, IOSvL2, etc…)

 

Next Steps & Additional Resources

Your CML2 instance is now capable of labbing Cisco SD-WAN, however additional configuration is required to “provision” your SD-WAN nodes for fabric enrollment!

The next steps that you should immediately take include:

Planning and designing a topology that allows you to lab Cisco SD-WAN exam objectives as you do not want to continuously waste time on configuring control infrastructure

• Jebediah Casey @ Neckercube.com: “Cisco SD-WAN Basic Configuration Lab” – This is an excellent break down of core SD-WAN components and provides a sample topology

Configuring the viptela.serial file through your Cisco Smart Account to enable wEdge SD-WAN fabric enrollment with vBond & vManage

• UPDATE 25 SEPT 2021: I have uploaded a viptela.serial file to my GitHub [link] to enable self-hosted labbing where you do NOT have a smart account, please refer to the below link for how to import and use the serial file.
• Brad Searle @ CodingPackets.com: “Cisco SD-WAN Self Hosted Lab Part 1” – provides a walkthrough for this entire step

 

Known Issues and Fixes

Please note that as this is an unsupported extension to CML2 functionality these fixes are not guaranteed to work.

Submit any additional issues via email:

• To: admin[at]ether-net[dot]com
• Subject: CML2 SDWAN New Known Issue: <ISSUE SHORT DESCRIPTION>
• Content: Include steps to reproduce, screenshots and information, resolution steps (if known)

All submitted issues that have been verified will be credited in subsequent revisions of this document.

---

Issue #1: “Failed to start node <NAME>” Unable to define node (Unable to clone image)” is displaying when I try to start a vBond/vEdge, vManage or vSmart node.

Example screenshot of issue displayed

 

You have incorrectly defined your node – this typically happens when the following settings are incorrectly configured:

• General > Nature is not “router”
• Linux Native Simulation > Simulation Driver is not “server”
• Linux Native Simulation > Boot Disk has a value set when the field should be empty

 

Issue #2: My node icons are hung on “Started” and are not transitioning to “Booted”

I think this happens when the boot timer timeout limit has not been hit and/or CML2 doesn’t understand how to parse boot output for a device. I fix this by lowering the boot timer timeout limit. Yet to experiment with boot options.

 

Issue #3: vManage does not give me the option to mount “hdb” upon first boot.

This can happen if:

1. You do not define the data disk as “IDE” in your node definition. Go back to your vManage node definition > Linux Native Simulation > Disk Driver : set to “IDE”.
2. You have defined the data disk as “IDE” but you have configured a Boot Disk instead of a Data Disk. Go back to your vManage node definition > Linux Native Simulation > Remove any configuration in the “Boot Disk Size” field and then enter a value into the “Data Disk Size”.

Issue #4: The node’s interfaces in Lab Manager are inconsistent with the “show interface | tab” output in Viptela OS.

You have not configured the interfaces correctly – go back to the section “Creating Cisco SD-WAN Custom Node Definitions” and follow the interface configuration EXACTLY.

 

Issue #5: vManage web interface is not working despite the fact that I have allocated it enough resources

You likely did not boot the vManage device first when turning on your lab, please refer to the “Recommended Node Spin Up Sequence” section.

 

Issue #6: vManage web interface has started to time out after I have added <X> devices to the SD-WAN fabric

You may have one of two issues:

1. Enough resources have not been provisioned to the host device; or
2. You may need to allocate additional resources to vManage (CPU and/or RAM).

 

Issue #7: vManage is not processing configuration changes and/or is randomly losing statistical information

Assuming your CML2 VM has not run out of storage space, check the vManage database disk space consumption as you’re most likely at capacity.

To resolve for 1 vManage node:

1. Power down the node
2. Wipe the node
3. Expand the data drive capacity in your lab workbench under its properties

To resolve for ALL vManage nodes in your CML2 instance:

1. Power down all vManage nodes
2. Wipe all vManage nodes
3. Expand the data drive capacity in the vManage node definition

 

Issue #8: IOSvL2 switches are INCREDIBLY laggy when used as a L3 site core switch in an SD-WAN lab despite the CML2 VM having plenty of available resources

Root cause unknown.

If your IOSvL2 switch is constantly spiking at 100% CPU utilization then extract its config, shut down the node and wipe it, and then double resources allocated to it.