Skip to content

F #-: VRouter: Management NIC in separate MGMT VRF - #419

Open
netaviator wants to merge 8 commits into
OpenNebula:masterfrom
netaviator:pr/vrouter-mgmt-vrf
Open

netaviator wants to merge 8 commits into
OpenNebula:masterfrom
netaviator:pr/vrouter-mgmt-vrf

Conversation

@netaviator

Copy link
Copy Markdown

OpenNebula marks one NIC of a Virtual Router as the management NIC. In OpenNebula it is set with VROUTER_MANAGEMENT = "YES" on the NIC of the Virtual Router template, and arrives in the VM context as ETH_VROUTER_MANAGEMENT=YES. one-context adds the default route of every NIC with ip route add, so the gateway of the first NIC wins and the default route of the management NIC collides with the one of the NIC that carries the traffic.

Move the management NIC into its own VRF (mgmt, routing table 1000), so its default route lives in a separate table and the main table stays free for the routed NICs. It is on by default. Opt out with ONEAPP_VNF_MGMT_VRF_ENABLED=NO. Without a NIC marked as management the module does nothing.

  • IPv6: moving a NIC into a VRF flushes its IPv6 addresses and routes, so the static address, the ULA and the default route are put back, inside the VRF table.
  • OneGate: the appliance talks to OneGate from the default VRF, which cannot reach hosts behind the management NIC once it is in the VRF. A policy rule sends the traffic for the OneGate address (IP literals only) to the VRF table.
  • Other NICs: with the management gateway in the VRF, the gateways of the other NICs (IPv4 with ETH_METRIC, IPv6) are installed in the main table, as one-context would have done.
  • tcp_l3mdev_accept and udp_l3mdev_accept are set, so sshd and other listeners of the default VRF accept connections on the management NIC.

Values that end up in a shell command (gateways, metrics, IPv6 addresses) are checked; an invalid one is logged and skipped.

The NIC that OpenNebula marks as VROUTER_MANAGEMENT moves into a VRF, so its default route no longer collides with the one in the main table. Opt out with ONEAPP_VNF_MGMT_VRF_ENABLED=NO.

Signed-off-by: Jan Gilla <jgilla@level66.network>
Moving a NIC into a VRF flushes its IPv6 addresses and routes. Put the static address, the ULA and the default route back, inside the VRF table.

Signed-off-by: Jan Gilla <jgilla@level66.network>
The appliance talks to OneGate from the default VRF, which cannot reach
hosts behind the management NIC once it is enslaved. Add a policy rule
that sends the traffic for the OneGate address (IP literals only) to the
VRF table, and remove it again at cleanup.

Signed-off-by: Jan Gilla <jgilla@level66.network>
one-context adds the default route of every NIC with 'ip route add', so
the gateway of the first NIC (the management one) wins and the others are
never installed. With the management gateway in the VRF table, apply the
gateways (IPv4 with ETHx_METRIC, IPv6) of the NICs that are not marked as
management in the main table.

Signed-off-by: Jan Gilla <jgilla@level66.network>
Describe the ONEAPP_VNF_MGMT_VRF_ENABLED attribute (on by default), what the
module configures and in which order, the accepted values and what it does
not cover.

Signed-off-by: Jan Gilla <jgilla@level66.network>
@roamingengineer311

Copy link
Copy Markdown
Contributor

@netaviator Hi.
I tested the management VRF in a three-NIC setup. Basic routing separation, OneGate connectivity, and boot behavior with VRF enabled and disabled work as expected.
I left three inline comments about issues that affect existing routing behavior: lost management-interface static routes, missing default routes after disabling VRF through re-contextualization, and a duplicate service default route. Could you please take a look?

@netaviator

Copy link
Copy Markdown
Author

Hi @roamingengineer311,

looks like I do not see your inline comments. Any chance you can provide them?

Best
Jan

Comment thread appliances/VRouter/MgmtVRF/main.rb Resolved
Comment thread appliances/VRouter/MgmtVRF/main.rb Resolved
Comment thread appliances/VRouter/MgmtVRF/main.rb Resolved
Moving a NIC into the VRF drops its static routes from the main table, and deleting the VRF drops the routes of the VRF table. Put the ETH<n>_ROUTES of the management NIC into the VRF table. On stop, give the default routes of all NICs, the IPv6 addresses and the ETH<n>_ROUTES of the management NIC back to the main table, since one-context does not run again. A default route is only added when the NIC has none, so one that one-context installed (for example with a metric) is kept.

Signed-off-by: Jan Gilla <jgilla@level66.network>
@netaviator

Copy link
Copy Markdown
Author

Thanks for testing this and for the detailed steps. I pushed 3ece29c, which addresses the three points:

  • Static routes: the ETH<n>_ROUTES entries of the management NIC are now added to table 1000 after the NIC moves into the VRF.
  • YES to NO by re-contextualization: when the service stops, the default routes, the IPv6 addresses and the static routes of the management NIC go back into the main table, and so do the default routes of the other NICs where they are missing.
  • Duplicate default route: a default route is only added when the NIC has none in the main table, so a route that one-context installed (for example metric 1 onlink) is kept.

I checked the first two cases and the existing default route on a VM with a management NIC and a second NIC. The unit specs cover the rest. The README describes the new behavior.

@roamingengineer311 roamingengineer311 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I retested the updated image and confirmed that the static route and re-contextualization issues are fixed. I left one follow-up comment about duplicate default routes still appearing after a service restart. Could you please take a look?

Comment thread appliances/VRouter/MgmtVRF/main.rb Resolved
…MgmtVRF

The bash helper runs scripts with pipefail. The check for an existing default route ended in grep -q, which exits at the first match while ip route show still writes. ip then dies of SIGPIPE, the pipeline fails with a default route in place, and a second default without metric was added next to the one of one-context. It happened in about half of the restarts in a test VM, depending on timing. The check now lets grep read the whole output.

Signed-off-by: Jan Gilla <jgilla@level66.network>

@roamingengineer311 roamingengineer311 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. I tested the latest change. No further concerns from my side.

… host name

A host name in ONEGATE_ENDPOINT got no policy rule, so the default VRF
could not reach OneGate behind the management NIC. The name is now
resolved through the VRF (falling back to the default VRF) once the NIC
and its routes are in place, and a rule is added for every IPv4 and IPv6
address. Stop and start delete every rule of preference 100, since the
addresses behind a name can change between runs.

Signed-off-by: Jan Gilla <jgilla@level66.network>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants