IP Fabric Netbox Plugin vs Device Library Import (UPDATED)

The very talented Alex Gittings from IP Fabric has helped me to understand how to get around the capitalisation issue. In the transform maps you can create a jinja2 template that capitalises a field. So if applied in the device - DCIM|manufacturer map, with source field as vendor and target field as name, it will apply all vendor names as capitals. The template would be {{ object.vendor.capitalized() }}

This post describes why the IP Fabric plugin for netbox fails when used in conjunction with the netbox device type library importer.

What are the three components?

• IP Fabric – Commercial software automates network discovery and assurance
• Netbox – Open Source network source of truth
• Netbox Plugin for IP Fabric – The add on to netbox that eases the ingestion of IP Fabric data into netbox
• Device Type Library – A community created repository of device templates for netbox
• Device Type library importer – Python code that automates the bulk import of the above into netbox

What is the Problem?

This post is about a complex problem. If you just want to import device types into netbox so nearly every network device has a template readily available, that works fine. If you want to import the devices IP Fabric has found into netbox, that works fine. If you attempt to do both tasks, it is not fine.

What Happens?

Both the IP Fabric plugin and the device type library importer are intolerant to situations I think they should be able to handle. For example if you import thousands of templates and there is an issue with just one, everything fails. In my case if you attempt to import a device type and one exists (but has slightly different name/case) then the process fails. My tip here is to delete all device types if possible before importing.

However, once that process is complete you have a netbox instance full of templates for router and switches etc. An example template is the Juniper MX960. This has fields:

model: MX960
slug: juniper-mx960

There is a reason I’m showing those and it is the cause of the woes. In fact I’ll show you what IP Fabric has found which is also some big Juniper routers:

IP Fabric is wonderfully API first and everything can be seen as JSON output. So here is the above in native format:

"platform":"mx","model":"mx960","vendor":"juniper","family":"junos"

At this point those with a strong coding background may already have a hint at why I have a chunk of my life I’ll never get back. When the IP Fabric ingest into netbox takes place, it fails. Every time. It provides a teasingly useful error message:

However, if you look at your netbox’s device type library you will indeed see there is a device matching the above. So the error message is correct. This would be fine if it wasn’t for the fact that having device types that match incoming data is very normal. It should simply say “I see the device type is there, so I’ll use that one”. If there is no device template it will say “I can’t see a device type entry so I’ll create one”. Instead the error message is saying “I can see there is a matching device type and I’m going to halt an ingest process that has been running for an hour. screw you”.

Troubleshooting

I’m using an instance running on docker. A similar process should be available for the traditional installs. Hopefully with this post you won’t need either. In Docker if you change directory to the netbox docker install you can run the following to get a realtime view of the log output (kind of tail -f on a syslog file).

docker compose logs --follow --tail 0

Below is an example successful message for a device being imported into netbox:

netbox-worker-1 | Creating vrf
netbox-worker-1 | {'hostnameOriginal': 'edge-fireawall1', 'fqdn': 'edge-fireawall1', 'sn': 'CZ2319A2222', 'loginType': 'ssh', 'taskKey': '136772bb-c098-4332-a0c5-f35958a50ffe', 'mac': None, 'rd': None, 'memoryUsedBytes': 1803550720, 'hostname': 'edge-fireawall1', 'id': '1372393001', 'hostnameProcessed': 'edge-fireawall1', 'uptime': 1843800, 'slug': None, 'loginIp': '10.2.3.42', 'siteName': 'Portsmouth', 'tsDiscoveryEnd': 1729731953056, 'secDiscoveryDuration': 45.402, 'processor': None, 'model': 'srx345', 'objectId': None, 'tsDiscoveryStart': 1729731907654, 'devType': 'fw', 'reload': '0x1:power cycle/failure', 'version': '22.5R3-S2.11', 'stpDomain': None, 'memoryUtilization': 41.99, 'image': None, 'vendor': 'juniper', 'configReg': None, 'icon': None, 'memoryTotalBytes': 4294967296, 'snHw': 'CZ1111AF0091', 'loginPort': 22, 'family': 'junos', 'platform': 'srx', 'domain': 'example.com'}
netbox-worker-1 | Device 62 out of 1142
netbox-worker-1 | Creating site
netbox-worker-1 | Creating manufacturer
netbox-worker-1 | Creating devicetype
netbox-worker-1 | Creating platform
netbox-worker-1 | Creating devicerole
netbox-worker-1 | Creating device

Cause?

The cause is because the device type library has things in capitals while IP Fabric loves the lowercase. Also the slugs used in the community library are chosen to ensure they never conflict so the MX960 would have junos-mx960. Here are the steps to misery:

1. The plugin code checks for a device type match using some level of case sensitivity. As mx960 doesn’t match the device type MX960 it decides there is no match.
2. The plugin creates a new entry for the device type which invokes a SQL INSERT command in the backend.
3. SQL complains that there is already an entry that matches (it isn’t case sensitive) and gives error messages like this:

postgres-1 | 2024-10-24 17:33:31.554 UTC [4079] ERROR: duplicate key value violates unique constraint "dcim_devicetype_unique_manufacturer_model "
postgres-1 | 2024-10-24 17:33:31.554 UTC [4079] DETAIL: Key (manufacturer_id, model)=(2, WS-C3650-48TS) already exists.

Solution/summary

There is no neat or quick way to resolve this. The data in the community library is not compatible with what IP Fabric creates. This is the case for all manufacturers. Aside from a revamp of the netbox plugin (I’ll try) the only way to proceed with the above combination of tools is to manually edit the device types so that both the name and slug match what IP Fabric has. This isn’t just lowering the case (which could be done in a script) but also deleting parts of some of the slugs. If you are lucky and have a small number of types on your network, this is easy. If you have a thousand devices types, hire an intern.

After doing the above I have a full device type library in netbox and can ingest more than a thousand devices from IP Fabric without errors.