Introduction
A really important design consideration when moving towards an automated network is the Single Source of Truth. The single place that holds a certain set of data. The data set is never copied anywhere but instead is referenced from anywhere that needs it. See why this is important here.
In the case of a classic network department looking after switches and routers a key SSoT is the database of devices and their details. For a HPE shop this could be IMC.
To run an ansible playbook you need a host file. This could be as simple as a list of switch names but can include variables and other details. This host file can be a truly dynamic file meaning it is created/referenced at runtime in some way. That is an advanced topic and isn’t optimal in every environment. Sometimes the generation of that file can take time which adds considerable latency to a simple ansible playbook but in some environments the hosts changes so often that the only way to ensure you are hitting the right network devices is to check the SSoT at the start of each task. More discussion here.
At the other end of the spectrum is the totally static/manual host file. The one that a human creates by exporting devices from a database or editing based on knowledge. This is very rarely a good idea. In some cases it can lead to disaster. If there is an assumption that the host file is accurate then devices added since its creation will not get touched. It is also inefficient to have a manual step and kind defeats the object of automation.
As with most of these issues there is a middle way and it will work well in a lot of network teams. The dynamically created static host file.
Given the situation where devices are added/removed/altered on a less than daily basis a host file created at runtime would look the same all day long. If that file was created overnight to reflect any recent changes then it would be fairly accurate for 24 hours and serve the system well in terms of speed of playbook execution. The host file could be re-created with a playbook execution (or manually edited) should an engineer want to update the system sooner (e.g. during device install). An automated network device install consisting of config generation and SSoT update could call this script to update the host file.
The advantage of this middle way is that the host file remains accurate since it is based on the SSoT. As soon as engineers start editing host files you firstly get errors and secondly have missed the point of network automation.
Playbook Details
The playbook to create this file contains two main task sections
- Getting the data from IMC via RESTful API
- Extracting relevant facts and writing to a text file
The first of these uses the uri module which is part of the base ansible install. This module can be used for any type of basic HTTP interaction but in this case we are interacting with a RESTful API. Find out a little about the IMC API implementation here.
In really basic terms the uri module sends a question via a HTTP GET message to IMC and returns the answer as structured data. We are able to request the response be sent in JSON which means the information sent back is immediately available to ansible tasks as variables. By default (e.g. if you paste the URLs into a browser) you get XML which while a standard way to respond from these APIs it requires additional steps before the data is usable. See future posts on Aruba Airwave to see why IMC’s JSON response is a blessing.
Here is an example of what you get when you ask for a list of devices:
{
"device": [
{
"id": "2",
"label": "switch018",
"ip": "10.4.1.1",
"mask": "255.255.255.0",
"status": "1",
"statusDesc": "Normal",
"sysName": " switch018",
"contact": "itsupport@constantpinger.com",
"location": "Dirty Wiring Closet",
"sysOid": "1.3.6.1.4.1.11.2.3.7.11.117",
"sysDescription": "HP J9565A Switch 2615-8-PoE, revision A.15.16.0008, ROM J.14.08 (/ws/swbuildm/rel_orlando_qaoff/code/build/lgr(swbuildm_rel_orlando_qaoff_rel_orlando)) (Formerly ProCurve)",
"devCategoryImgSrc": "switch",
"topoIconName": "iconswitch",
"devPingState": "1",
"categoryId": "1",
"symbolId": "1004",
"symbolName": " switch018",
"symbolType": "3",
"symbolDesc": "",
"symbolLevel": "3",
"parentId": "1003",
"typeName": "HP 2615-8-PoE Switch",
"mac": "d8:9d:67:ff:33:33",
"link": {
"@op": "GET",
"@rel": "self",
"@href": "https://imc.constantpinger.com/imcrs/plat/res/device/2"
}
},
The response is stored as a variable using the normal register command. We could use this information straight away if we only wanted a host file with names and perhaps device types (e.g. switch differentiated from routers). The second task could be skipped and the output from the first be used to fill out the host file.
However, I’ve added the second API call to demonstrate a useful concept (using dynamically sought information to create a list of information to ultimately ask and use) and also to demonstrate IMCs second blessing (URLs returned that point to related information). Using the second task I’m able to populate the host file with facts such as the location and firmware version. In my example this allows different groups of devices that are in the DC compared to in the distribution/edge. In a full ansible implementation you would be advised to populate a VAR file. Remember: playbook is a demo.
Some things to note from the first two tasks:
- The URL contains a size argument that can be made small during testing for a quick response. A thousand device database takes a few minutes to run through with this playbook. Be careful not to return just one item. Ansible can sometimes do strange things when it expects a list but gets a single item. More on that in a future post on wantlist.
- The uri module argument validate_certs: no should be edited to yes in production for security.
- The second task has a URL pointing to {{(item.link|dict2items)[1].value}} while looping through each item in the list returned in the first task. Each item in the first task list (basic details of IMC device) has a link property (see above). That is itself a small dictionary with the second item being a URL to get the detailed facts about the device. This shows that in my playbook I am using the first task purely to deliver a URL for each device. All the details used come from the second task. While in theory you could go direct to the detailed info by crafting these URLs, this only works if you know all the IDs. If you request details of a device with a non existent URL the process fails. So in short the first tasks ensures you get a list of URLs that actually exist and the second task gathers all the details.
- The dict2items filter in the second tasks URL is just one way to extrapolate the value of a specific element. It returns a list from the values of the dictionary. There are other ways to extract the value, this is put in as a demo. If you ever need to loop through a dictionary this is the way to achieve that.
The second part of the tasks writes lines to the text file based upon all the data gained during the second task. I’ve simplified it by using the same ansible module (lineinfile) although this can be made more elegant with thte use of other modules such as blockinfile.
The lineinfile module is quite powerful and few of the many features are used here. One to point out is the create: yes parameter. This creates the file if it isn’t present. If the file exists it does nothing. If this isn’t present then the default behaviour is to fail the task. I’ve added it to the first task in the section but you could chose to add it to all tasks calling lineinfile to ensure it is always created.
I’ve added the path as a play level variable ({{path}}) and directly. Both are correct syntax but this demonstrates bad coding. All tasks should point to a single variable to ensure you write to the same file.
If you were to use this host file directly this would point to the host file used by other playbooks. If you want to experiment or keep a record you might want to use set_fact to create a variable that includes a date-stamp.
The tasks are fairly simple to follow. The first writes the static sections as required by ansible if you want to group devices. The next chunk filters through all the devices and writes a line if the when statement matches (e.g. if the switch model matches). The insertafter: ‘^[wifi]’ parameters insert the line directly below the section headers (pushing the existing lines down one). This is why you’ll see the switches in reverse alphabetical order.
It is worth pointing out the task add_MISC_to_hosts which is slightly different to the previous ones. All of the lineinfile tasks above search for a string being present in the device data. The misc tasks checks that certain things are not present. This is an example of where the logic needs to be in the right order. If this tasks was at the top of the section then no devices would appear under the various headings. They would all be under [misc]. Note also that the search is case sensitive. In this playbook the conditional could be left of which would allow all devices to be ‘caught’. It is put here purely for demonstration purposes.
Ansible doesn’t need a blank line after each section of hosts it just assumes the end of section when it reads a [ character. As a human it is much easier to review the file with a space between sections hence the very last tasks. Note that if you insert blank lines in the first lineinfile task they are erased by subsequent tasks that take over blank lines they find in the right location rather than inserting a new line every time (i.e. they only push the file contents down one line if the line under insertafter isn’t blank). With a bit of work this could be more elegant by including something in the first static text task.
FULL PLAYBOOK
---
# Playbook creates an ansible host file by askinging IMC for the list of all devices which includes a URL for each pointing to full device details then extracting values from that devices's info
# example command line: ansible-playbook -i localhost, ~/ansible/complete/askIMC_HostFileCreator.yml
- name: get devices from IMC
hosts: all
strategy: free
gather_facts: no
connection: local
no_log: true
vars_prompt: #variables that you want to ask the user at run time
- name: "username"
prompt: "username?"
private: no
- name: "password"
prompt: "password?"
private: yes
vars: #variables that are fixed and relevant to the whole play. Some aren't used for this example but are standard throughout the playbook series
path: ~/script_output/IMC_hosts.txt
password: {{password}}
username: {{username}}
ansible_user: "{{username}}"
ansible_password: "{{password}}"
login:
host: "{{inventory_hostname}}"
username: "{{ username }}"
password: "{{ password }}"
############################################################
tasks:
- name: askIMC_list_devices
uri:
url: https://imc.constantpinger.com:443/imcrs/plat/res/device?resPrivilegeFilter=false&start=0&size=99999&orderBy=label&desc=false&total=false&exact=false # the size variable can be set very low for testing to reduce the size of the data returned
method: GET
user: "{{username}}"
password: "{{ password }}"
headers:
accept: "application/json"
content-type: "application/json"
status_code: 200
body_format: json
validate_certs: no
return_content: yes
register: result
- name: askIMC_device_details
uri:
url: "{{(item.link|dict2items)[1].value}}" #use URLs extracted from the first search in askIMC_list_devices
method: GET
user: "{{username}}"
password: "{{ password }}"
headers:
accept: "application/json"
content-type: "application/json"
status_code: 200
body_format: json
validate_certs: no
return_content: yes
register: result2
loop: "{{ result.json.device }}"
########### All info gathered and stored in variable result2 - part2 populates the text file ###########
- name: create unpopulated host file
lineinfile:
create: yes
path: ~/script_output/IMC_hosts.txt #change to location needed
line: "{{item}}"
with_items:
- '[routers_switches:children]'
- 'switches'
- 'routers'
- '[routers]'
- '[routers:vars]'
- 'router=true'
- 'os=comware7'
- '[switches:children]'
- 'comware_5130'
- 'procurve'
- 'walljacks'
- 'aruba'
- '[comware_5130]'
- '[comware_5130:vars]'
- 'router=false'
- 'os=comware7'
- '[arubaos]'
- '[arubaos:vars]'
- 'router=false'
- 'os=procurve'
- '[walljacks]'
- '[procurve]'
- '[procurve:vars]'
- 'datacentre=false'
- 'router=false'
- '[dc_leafs]'
- '[dc_leafs:vars]'
- 'router=false'
- 'datacentre=true'
- '[wifi]'
- '[wifi:vars]'
- 'datacentre=true'
- '[misc]'
- name: add_5130_non_DC_to_hosts
lineinfile:
path: "{{path}}"
line: "{{item.json.label}} typeName=\"{{item.json.typeName}}\" version=\"{{item.json.version}}\""
insertafter: '^\[comware_5130]'
when:
- item.json.typeName is search("5130")
- item.json.location is not search("dc2")
- item.json.location is not search("DC2")
- item.json.location is not search("DC1")
- item.json.location is not search("dc1")
loop: "{{ result2.results }}"
- name: add_walljacks_to_hosts
lineinfile:
path: "{{path}}"
line: "{{item.json.label}}"
insertafter: '^\[walljacks]'
when: item.json.typeName is search("NJ5000")
loop: "{{ result2.results }}"
- name: add_arubaOS_to_hosts
lineinfile:
path: "{{path}}"
line: "{{item.json.label}} version=\"{{item.json.version}}\""
insertafter: '^\[arubaos]'
when: "item.json.typeName == 'Aruba Stack 2930F'"
loop: "{{ result2.results }}"
- name: add_wifi_controllers_to_hosts
lineinfile:
path: "{{path}}"
line: "{{item.json.label}} typeName=\"{{item.json.typeName}}\" version=\"{{item.json.version}}\""
insertafter: '^\[wifi]'
when: 'item.json.typeName is search("Aruba 7")'
loop: "{{ result2.results }}"
- name: add_procurve_to_hosts
lineinfile:
path: "{{path}}"
line: "{{item.json.label}} typeName=\"{{item.json.typeName}}\" version=\"{{item.json.version}}\""
insertafter: '^\[procurve]'
when:
- item.json.typeName is search("HP 19") or item.json.typeName is search("HP 26") or item.json.typeName is search("HP 29") or item.json.typeName is search("25") or item.json.typeName is search("HP 540")
- item.json.location is not search("dc2") or item.json.location is not search("DC2") or item.json.location is not search("DC2") or item.json.location is not search("dc1")
loop: "{{ result2.results }}"
- name: add_routers_to_hosts
lineinfile:
path: "{{path}}"
line: "{{item.json.label}} version=\"{{item.json.version}}\""
insertafter: '^\[routers]'
when:
- item.json.typeName is search("5900") or item.json.typeName is search("5930") or item.json.typeName is search("5510")
- item.json.label is search("str1") or item.json.label is search("str0")
loop: "{{ result2.results }}"
- name: add_DC_leafs_to_hosts
lineinfile:
path: "{{path}}"
line: "{{item.json.label}} version=\"{{item.json.version}}\" location=\"{{item.json.location}}\""
insertafter: '^\[dc_leafs]'
when:
- item.json.typeName is search("5900") or item.json.typeName is search("5130") or item.json.typeName is search("5500")
- not (item.json.label is search("str1") or item.json.label is search("str0"))
- item.json.location is search("dc2") or item.json.location is search("DC2") or item.json.location is search("DC2") or item.json.location is search("dc1")
loop: "{{ result2.results }}"
- name: add_MISC_to_hosts
lineinfile:
path: "{{path}}"
line: "{{item.json.label}} typeName=\"{{item.json.typeName}}\" version=\"{{item.json.version}}\" location=\"{{item.json.location}}\""
insertafter: '^\[misc]'
when:
- item.json.typeName is not search("5900")
- item.json.typeName is not search("5930")
- item.json.typeName is not search("5130")
- item.json.typeName is not search("5510")
- item.json.typeName is not search("NJ500")
- item.json.typeName is not search("2930")
- item.json.typeName is not search("HP 26")
- item.json.typeName is not search("HP 29")
- item.json.typeName is not search("HP 25")
- item.json.typeName is not search("HP 540")
- item.json.typeName is not search("HP 19")
- item.json.typeName is not search("Aruba 7")
loop: "{{ result2.results }}"
########### add blank lines for human readability ###########
- name: blank line before [arubaos] section1
lineinfile:
path: "{{path}}"
line: ' '
insertbefore: '^\[arubaos]'
- name: blank line before [comware_5130] section
lineinfile:
path: "{{path}}"
line: ' '
insertbefore: '^\[comware_5130]'
- name: blank line before [walljacks]
lineinfile:
path: "{{path}}"
line: ' '
insertbefore: '^\[walljacks]'
- name: blank line before [dc_leafs]
lineinfile:
path: "{{path}}"
line: ' '
insertbefore: '^\[dc_leafs]'
- name: blank line before [routers]
lineinfile:
path: "{{path}}"
line: ' '
insertbefore: '^\[routers]'
- name: blank line before [routers:vars]
lineinfile:
path: "{{path}}"
line: ' '
insertbefore: '^\[routers:vars]'
- name: blank line before [switches:children]
lineinfile:
path: "{{path}}"
line: ' '
insertbefore: '^\[switches:children]'
- name: blank line before [procurve]
lineinfile:
create: yes
path: "{{path}}"
line: ' '
insertbefore: '^\[procurve]'
- name: blank line before [wifi]
lineinfile:
path: "{{path}}"
line: ' '
insertbefore: '^\[wifi]'
- name: blank line before [MISC]
lineinfile:
path: "{{path}}"
line: ' '
insertbefore: '^\[MISC]'
Constantpinger 