Same IP address for different machines confuses Ansible

I was dealing with multiple GCP projects in my Ansible environment. These projects were defining multiples VPCs which operates on the same CIDRs. Two machines were attributed a same IP address, which is normally not an issue since they don't communicate to each other. However, Ansible is accessing both.

When using SSH connections with Ansible, Ansible sets a control socket for each machine, allowing the connection to be used multiple times to increase performances (see Pipelining). This control socket is defined as a file, which path is calculated from the host, port and user parameter.

This works well when accessing directly the machines. However, when using jump host or bastion hosts, one single IP address can match different machines, depending on which jump host is targeted. The problem is, when we want to target these machines, a single control socket is defined, which effectively target only one host (the first one Ansible connects to). You end up with playbooks being played on a completely wrong host, which can be catastrophic.

The solution

The solution here is to redefine how control path is set. We cannot do it from Ansible since it does not allow variable usage in the control path (and since Ansible 2.4, the parameter seems to have vanished from the documentation, even if still editable. The only remaining option is the control path dir). Therefore we have to edit the ansible_ssh_common_args option. If you use jump host, you probably already use it since this is were you defined the ProxyCommand option. You need to add the ControlPath option is configure it to have a unique identifier for your host that share the same IP. In my case, I set this:

-o ControlPath=~/.ansible/cp/{{ name ~ '-' ~ ansible_user }}

name is a variable which contains the name of the machine (you can also use inventory_hostname). In my environment, I know that machine name are unique, and since I do not have multiple SSH on a machine, I do not need to set the port.

This is how my ansible_ssh_common_args looks like:

ansible_ssh_common_args: >-
  -o ProxyCommand="gcloud compute start-iap-tunnel {{ name }} %p --listen-on-stdin --project={{ project_id }}
  --zone={{ instance_zone }} --verbosity=warning" -o ControlPath=~/.ansible/cp/{{ name ~ '-' ~ ansible_user }}

Now, I can access both the machines at the same time.