CLI Hub & Spoke in Azure

published: July 27, 2022

Intro

A few weeks ago I delivered a session at work about Azure, in which I created a simple Hub & Spoke network using Azure CLI. This was my first encounter with Azure so I believe is worth sharing the experience and saving my notes as well.

The goal of my presentation was to showcase how you can use Azure CLI to automate resources and connected them using a VNet Peering, so the network show here is simple, which at the same time is good if you are starting on the Azure world

Topology

On the diagram below you can see what we try to achieve, we have three VMs, each one on their own VNet and they are connected through a VNet Peering.

For this example, each VM has a Public and a Private IP, even though we can SSH directly to each VM through the internet, I want to connect the VMs via their private IP, this is possible with VNet Peering.

For this exercise I'm using debian since all I want is a VM with SSH to connect and test connectivity.

An important point to consider is that all resources in a VNet can communicate directly to the internet by default, this is only for outbound direction. So if all the VMs are on inside the same VNet then no VNet Peering is needed.

Before start

If this is the first time you are working with azure, the first step is to login to azure, to do this just type az login on your terminal. This is assuming you have azure cli tools installed on your computer. If you don't have it, see these instructions on how to get it

This will open a window in your browser and from there you can login.

tip, use --help with the azure cli when you don't know how to configure something or what options you have available. This was really useful.

I will keep the commands on the post as a template, if you want to see full guide I used, feel free to check out my readme, there you can copy all the commands and you should be able to create the same network I used.

Resource Groups

Now is time to create a resource group, which is basically a container to manage all your azure resources, like VMs. You can do it with the command below. In my case I used techTalk as the name of my resource group.

az group create --name techTalk --location northeurope

In my case I used northeurope as is the closest to my location, but you can chose the one that fits better for you. Use az account list-locations -o table if you want to know the full list of locations.

VNets

Next, we create our three VNets; hub, spoke1 and spoke2. Below is the template I used to create all VNets.

az network vnet create \
  --name <VNET_NAME> \
  --resource-group techTalk \
  --subnet-name hub-subnet \
  --address-prefixes <VNET_PREFIX>/16 \
  --subnet-prefixes <SUBNET_PREFIX>/24

Remember that our VNETs use the following IP addresing squeme.

VNet name	VNet prefix	Subnet Prefix
hub-vnet	10.0.0.0/16	10.0.1.0/24
spoke1-vnet	10.1.0.0/16	10.1.1.0/24
spoke2-vnet	10.2.0.0/16	10.2.1.0/24

Verify the VNets were created correctly using az network vnet list -o table.

╰─ az network vnet list -o table
Name         ResourceGroup    Location     NumSubnets    Prefixes     DnsServers    DDOSProtection
-----------  ---------------  -----------  ------------  -----------  ------------  ----------------
hub-vnet     techTalk         northeurope  1             10.0.0.0/16                False
spoke1-vnet  techTalk         northeurope  1             10.1.0.0/16                False
spoke2-vnet  techTalk         northeurope  1             10.2.0.0/16                False

Virtual Network Gateway

Usually the Virtual Network Gateway will provide us connectivity with our sites or users, in our case we are only to connect spokes between them as next hop.

First create a subnet and add it to our current hub-vnet, remember we can have multiple subnets inside of a VNet. In this case I called it GatewaySubnet.

az network vnet subnet create \
  --vnet-name hub-vnet \
  --name <SUBNET_NAME> \
  --resource-group techTalk \
  --address-prefix <SUBNET_PREFIX>/27

Then request a public IP for this gateway

az network public-ip create \
  --name <IP_NAME> \
  --resource-group techTalk \
  --allocation-method Dynamic \
  --sku Basic

Lastly, create the actual gateway. This will take a while, so I used the --no-wait option to just create the request but no wait for the answer from azure.

az network vnet-gateway create \
  --name <SUBNET_NAME> \
  --location northeurope \
  --public-ip-address <IP_NAME> \
  --resource-group techTalk \
  --vnet <VNET_NAME> \
  --gateway-type Vpn \
  --sku Standard \
  --vpn-type RouteBased \
  --no-wait

The gateway could take up to 40 minutes to boot up. Also, if you are learning, be sure to remove it if you are not using it, it will use your credit for just being alive. VMs could be left, but this one will cost you, so remove it once you are done.

host routes

For the spokes to communicate between each other, is needed to create a host route. This is a normal and traditional behaviour in networking, but is just using the azure way.

First create a route table

az network route-table create \
  --resource-group techTalk \
  --name <ROUTE_TABLE_NAME>

Now create a route entry

az network route-table route create \
 --name <ROUTE_ENTRY_NAME> \
 --resource-group techTalk \
 --route-table-name <ROUTE_TABLE_NAME> \
 --address-prefix <REMOTE_NETWORK>/24  \
 --next-hop-type VirtualNetworkGateway

And finally associate the route table to a subnet

az network vnet subnet update \
  --vnet-name <VNET_NAME>\
  --name <SUBNET_NAME> \
  --resource-group techTalk \
  --route-table <ROUTE_TABLE_NAME>

As in traditional routing, you need to create an entry for the trafic going in the other direction. This is how I see networking in the cloud, same concepts, just different implementation.

Remember you can see the full commands I used here

Virtual Machines

Finally we can create the virtual machines that will function as our hosts for this experiment.

As you can see, I tried to be very price observant, I used the smallest VM size and used all options available to remove stuff in case I need to remove the VM, like disks.

az vm create \
  --resource-group techTalk \
  --name <VM_NAME> \
  --image debian \
  --generate-ssh-keys \
  --public-ip-sku Basic \
  --size Standard_B1ls \
  --vnet-name hub-vnet \
  --subnet hub-subnet \
  --nic-delete-option Delete \
  --data-disk-delete-option Delete \
  --os-disk-delete-option Delete \
  --no-wait

Something I like about azure, the first time I used --generate-ssh-keys, it created a pair of keys for me and store them on my ~/.ssh directory. For subsequent usage, it re-use the previous keys generated. For development that's very handy.

For production I recommend to use secrets, env vars or an external service like vault.

VNet Peering

Lastly, we need to create the peering that will allow us to connect our VMs on different VNets via their private IP. As I commented at the beginning they can connect through the internet with the options we used on the VMs, but since our goal is to communicate via their private IP, the peering is needed.

The peering must be created on the hub and on every spoke we want to communicate with. This means, we need one peering from hub-to-spoke1 and hub-to-spoke2.

In the hub we use this command.

az network vnet peering create \
    --resource-group techTalk \
    --name <HUB_PEERING_NAME> \
    --vnet-name <HUB_VNET_NAME> \
    --remote-vnet <SPOKE_VNET_NAME> \
    --allow-vnet-access \
    --allow-gateway-transit \
    --allow-forwarded-traffic

And on the spokes we used.

az network vnet peering create \
    --resource-group techTalk \
    --name <SPOKE_PEERING_NAME> \
    --vnet-name <SPOKE_VNET_NAME> \
    --remote-vnet <HUB_VNET_NAME> \
    --allow-vnet-access \
    --use-remote-gateways

Notice the main difference between both command is the use of --allow-gateway-transit, the hub vnet is the only that has in this example a gateway configured.

Connect to the VMs

At this point, we should have everything up and we should be ready to check if our experiment worked. In azure cli, when there is an error you can see it on the terminal as soon as the response is receive, so if you haven't see one, it should be working.

Our test, will be very straightforward, we will SSH our hub, once in the hub, we will SSH our spokes using their private IPs. This will guarantee we are using our peering and not going through the internet.

To know the IPs of our VMs we can use the following command.

az vm list-ip-addresses -o table -n hub
az vm list-ip-addresses -o table -n spoke1
az vm list-ip-addresses -o table -n spoke2

In my case, this is the output I got.

╰─ az vm list-ip-addresses -o table -n hub
VirtualMachine    PublicIPAddresses    PrivateIPAddresses
----------------  -------------------  --------------------
hub               20.223.247.160       10.0.1.4

╰─ az vm list-ip-addresses -o table -n spoke1
VirtualMachine    PublicIPAddresses    PrivateIPAddresses
----------------  -------------------  --------------------
spoke1            20.238.107.221       10.1.1.4

╰─ az vm list-ip-addresses -o table -n spoke2
VirtualMachine    PublicIPAddresses    PrivateIPAddresses
----------------  -------------------  --------------------
spoke2            20.238.86.124        10.2.1.4

As a recommendation, don't save these public IPs, these will be released by azure eventually given the options we are using.

SSH keys forwarding

To test our connectivity, first we need to forward our keys to our hub, remember we didn't configure an username and password, instead we used SSH keys. Technically, you can copy and paste your private key on the hub VM, but you can also forward your keys using SSH. If you are on Mac or Linux, you can do the following.

First add your keys to the ssh-agent

╰─ ssh-add ~/.ssh/id_rsa
Identity added: /Users/jillesca/.ssh/id_rsa (/Users/jillesca/.ssh/id_rsa)

Next, you can configure your SSH config to forward your keys in our ssh/config file, just add the IP of your hub VM

╰─ cat ~/.ssh/config
Host 20.223.247.160
  ForwardAgent y

Or you can just use the -A flag on SSH to bypass your config file. This is what I used in my case, since the public IP will change eventually.

If you want to learn more check out this article

Verification

Finally the time of truth, let's see if we can enter the VMs we just created and SSH into the spokes from our hub using our private keys.

╰─ ssh -A 20.223.247.160                                                                                                        ─╯
Linux hub 4.19.0-20-cloud-amd64 #1 SMP Debian 4.19.235-1 (2022-03-17) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Jul 26 20:56:18 2022 from 5.249.50.103

jillesca@hub:~$ ip a | grep 24
    inet 10.0.1.4/24 brd 10.0.1.255 scope global eth0

jillesca@hub:~$ ip route
default via 10.0.1.1 dev eth0
10.0.1.0/24 dev eth0 proto kernel scope link src 10.0.1.4
168.63.129.16 via 10.0.1.1 dev eth0
169.254.169.254 via 10.0.1.1 dev eth0

jillesca@hub:~$ ping  10.1.1.4 -c 2
PING 10.1.1.4 (10.1.1.4) 56(84) bytes of data.
64 bytes from 10.1.1.4: icmp_seq=1 ttl=64 time=1.26 ms
64 bytes from 10.1.1.4: icmp_seq=2 ttl=64 time=1.80 ms

--- 10.1.1.4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 2ms
rtt min/avg/max/mdev = 1.258/1.529/1.800/0.271 ms

jillesca@hub:~$ ssh 10.1.1.4
Linux spoke1 4.19.0-20-cloud-amd64 #1 SMP Debian 4.19.235-1 (2022-03-17) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Jul 26 20:49:00 2022 from 10.0.1.4

jillesca@spoke1:~$ who
jillesca pts/0        Jul 26 20:57 (10.0.1.4)

jillesca@spoke1:~$

From the output above you can see the following:

Line 1 we connect using the -A flag to forward our keys.
Line 12, we see our internal IP is 10.0.1.4/24
Line 15, we can see, we only know our IPs, no comments of our public and peering IPs. This is handled by azure.
Line 21, we have connectivity to our spoke
Line 30, we can ssh our spoke with our forwarded keys.
Line 41, we can verify that our SSH is initiated from 10.0.1.4

Conclusion

I hope this little experiment help you in someway. When I was preparing my talk, I have to dig a bit to find how to put several pieces together and test them. I will be happy to know if somebody could save some time by following my post. It was also a nice first approach to azure, I'm already thinking what could be the next experiment.

Additionally if you are thinking these are too many commands to write and apply every time you want to do a change, you are right, all of these commands are fine to use once or twice. The power of the cli comes with automation, for example as a part of your CI/CD pipeline you can create VMs, VNets, Peering, etc., then the cli is very handy for these cases since you can directly create an azure element.

Lastly, remember you can see the full example I used here