SevOne NMS Advanced Network Configuration Guide

using NMCLI

Execute the following steps.

1. U sing ssh , log into SevOne NMS appliance as root .

   $ ssh root@<NMS appliance>

2. Create bonded interface. For example, bond0.

   Example

   $ nmcli connection add type bond con-name bond0 ifname bond0 \

   ip4 10.168.116.5/22 gw4 10.168.116.1 ipv4.method manual \

   ipv6.method ignore \

   bond.options "mode=active-backup,miimon=100,downdelay=400,updelay=400"

    

   Connection 'bond0' (e8048f88-80e5-43c1-a981-3bcb9b8ccc69) successfully added.

   
   

3. Add the first interface to bond0 created above.

   Example

   $ nmcli connection add type bond-slave ifname ens160 master bond0

    

   Connection 'bond-slave-ens160' (fe926d34-0e12-46e3-a07c-fa78e5e0c8a3) successfully added.

   
   

4. Bring the connection up for the first follower interface.

   
   

   Example

   $ nmcli conn up bond-slave-ens160

    

   Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/51).

   
   

5. Set the first follower interface to the primary interface for the bond0.

   $ nmcli device modify bond0 +bond.options "primary=ens160"

    

   Connection successfully reapplied to device 'bond0'.

   
   

6. For each additional interface, add it as a follower and then, bring it up.

   $ nmcli connection add type bond-slave ifname ens33 master bond0

   Connection 'bond-slave-ens33' (073517f9-723a-4abe-a97e-fc10077f0ef3) successfully added.

    

   $ nmcli connection up bond-slave-ens33

   Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/51).

using NMTUI

1. Using ssh , log into SevOne NMS appliance as root .

   $ ssh root@<NMS appliance>

   
   

2. Enter NMTUI.

   $ nmtui

   

   
   

3. Select Edit a connection and click <RETURN>.

4. Navigate to <Add> and click <RETURN>.
   

5. Choose Bond from New Connection list.

   

6. Navigate to <Create> and click <RETURN>.
   

7. You are in Edit Connection to create your new bonded interface.

   

   Add a follower for every interface in the setup, change the mode of the interface, and adjust other settings.

8. When adding followers, choose Ethernet from New Connection list.

   

   
   

9. Add a proper Device name. The device name must be the interface name. For example, ens160, eth1, eth2.

   

10. Navigate to <OK> to save the configuration.

11. Select Activate a connection to confirm that all connections are active.

    

12. All active follower/bond interfaces show an asterisk ( * ) to the left of its name. If you do not see an asterisk, go to the connection and activate it.
    

    

13. Execute the following command to view the list of active network connections. All follower interfaces along with bond0 interface must appear in the list.

    Example

    $ nmcli connection

    NAME UUID TYPE DEVICE

    bond0 9aa751f6-828d-42dc-90ad-764ee6eb1b8f bond bond0

    ens160 e2f7df86-55c8-4227-a00d-0f048e030b1a ethernet ens160

    Wired connection 1 69c7ab6d-d760-3ba3-acfc-9ff7f05e6453 ethernet ens192

    Wired connection 2 3a9ca5e0-9c9e-3bee-9329-d3482093de2c ethernet ens224

    Ethernet connection 1 8331fa43-b8fb-4afe-95e8-a801508d3d8b ethernet --

    
    

14. Verify the configurations.

    The output in the examples below are for active-backup mode. The bonding options / configuration may vary for different deployments. It is recommended that the configuration is verified according to the mode selected.

    
    

    Examples

    $ cat /etc/sysconfig/network-scripts/ifcfg-bond0

    BONDING_OPTS="downdelay=400 miimon=100 mode=active-backup updelay=400"

    TYPE=Bond

    BONDING_MASTER=yes

    PROXY_METHOD=none

    BROWSER_ONLY=no

    BOOTPROTO=none

    IPADDR=10.168.116.5

    PREFIX=22

    GATEWAY=10.168.116.1

    DEFROUTE=yes

    IPV4_FAILURE_FATAL=no

    IPV6INIT=no

    NAME=bond0

    UUID=f3fa06e0-66b7-47d6-aff3-3a3db27ba596

    DEVICE=bond0

    ONBOOT=yes

    $ cat /proc/net/bonding/bond0

    Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

    Bonding Mode: fault-tolerance (active-backup)

    Primary Slave: ens160 (primary_reselect always)

    Currently Active Slave: ens160

    MII Status: up

    MII Polling Interval (ms): 100

    Up Delay (ms): 400

    Down Delay (ms): 400

    Slave Interface: ens160

    MII Status: up

    Speed: 10000 Mbps

    Duplex: full

    Link Failure Count: 0

    Permanent HW addr: 00:50:56:8c:11:13

    Slave queue ID: 0

    Slave Interface: ens33

    MII Status: up

    Speed: 1000 Mbps

    Duplex: full

    Link Failure Count: 0

    Permanent HW addr: 00:50:56:8c:2c:d0

    Slave queue ID: 0

    
    

15. From another appliance, verify that you are able to ping the IP address of bond0 created in the example above.

    $ ping 10.168.116.5

    PING 10.168.116.5 (10.168.116.5): 56 data bytes

    64 bytes from 10.168.116.5: icmp_seq=0 ttl=63 time=0.293 ms

    64 bytes from 10.168.116.5: icmp_seq=1 ttl=63 time=0.331 ms

    64 bytes from 10.168.116.5: icmp_seq=2 ttl=63 time=0.249 ms

    64 bytes from 10.168.116.5: icmp_seq=3 ttl=63 time=0.285 ms

    If the ping does not respond, restart the network service on the appliance you are bonding.

    $ systemctl restart network

Single point of access to NMS Cluster Graphical User Interface portal

A SevOne NMS cluster can be accessed using the IP address or FQDN of any of the member appliances in the cluster for the graphical user interface portal access. SevOne NMS administrators may want to provide access to the cluster using a single IP address or FQDN which points to a single peer - it may be the Cluster Leader peer or any other peer in the cluster. A VIP configuration on that specific pair ensures that access to SevOne NMS will always be maintained via the active polling peer.

Poll devices via a single IP address

Generally, the devices being polled on a peer pair require that the device is able to communicate with SevOne NMS via both, the primary and the secondary appliances of the pair. In some configurations, for example, where the user has strict access control required for the devices, it requires the device to have Access Control Lists (ACL) configured for both, primary and secondary NMS appliances. This increases the administrative overhead, especially if there are thousands of devices across the environment.

The user prefers to have the ability to poll devices on SevOne NMS peer using a single IP address, irrespective of the primary or the secondary appliance being the active poller at any given time. In such scenarios, a Virtual IP can be configured for the peer in SevOne NMS, and as long as the Virtual IP has network connectivity with the device on the required ports, polling can happen via the VIP.

Send Network Flows to SevOne NMS on a single IP address configured on Flow devices

Flow devices which send network flows to SevOne NMS require configuration of the device(s) to send flows to both, SevOne NMS primary and secondary appliances to ensure that flows are available when a failover happens.

To avoid the extra configuration and the extra bandwidth requirement to send the same flows to two different servers, the user may prefer to configure the flow device to send the flows to a single IP address. In such cases, the user may configure a VIP on SevOne NMS and configure the network flow device on their end to send the flows to the VIP address. No additional static route configuration is required on the NMS peer/DNC when devices are sending flows to the VIP as long as the user network allows the NetFlow traffic from the device to SevOne NMS peer appliances on the specific port configured for receiving flows on SevOne NMS.

Third-party Application Integration with SevOne NMS

The user may have third-party applications that integrate with SevOne NMS using the SevOne API. It may be a challenge for the user's third-party applications to be able to track the failure of SevOne NMS appliances in a pair, to continue to send the requests to the available appliance in case the active appliance were to fail or become unavailable.

The user may configure a VIP on the target NMS Peer pair and then the third-party application can be configured to send the API requests to the required VIP address of the peer.

Create Variables

Create variables by populating them in single-quotes, with the specific values that apply to your setup. These variables must be created and set on both the PAS and HSA.

Configure Virtual IP on Primary (PAS) and Secondary (HSA)

Perform the following steps on both, PAS and HSA appliances, to configure the Virtual IP on the NMS pair.

Configure Network Configuration Scripts for Network Connection to use Virtual IP

SevOne NMS requires that the default route is set via the Base IP address. It is not supported to have the default route to be configured over the Virtual IP address since all inter-peer communication in SevOne NMS happens via the Base IP address. Static routes may be configured for custom requirements - please refer to section Custom Static Route for Virtual IP.

Move and Link Updated Network Connection's Configuration Files

Update 'virtual_ip' for Current Peer (in the Config Database instance of the Cluster Leader)

To enable SevOne NMS to use the Virtual IP Address, update the virtual_ip column in the peers table for the pair. The update to the database must be performed from the Cluster Leader appliance by updating the config database instance.

Update MySQL Permissions (Optional)

If specific static routes are enabled for the pair during SevOne NMS Virtual IP configuration, MySQL permissions must be granted for IP address, $vip, to all pairs in the cluster. Execute the steps below.

1. Identify the IP Address of the Cluster Leader.

   $ ssh root@<PAS appliance>

   PAS$ cluster_master_ip=$(mysqlconfig -Ne \

   "select ip_normalize(ip) from net.peers where master = 1")

2. SSH into the Cluster Leader and execute SevOne-fix-mysql-permissions.

   PAS$ ssh $cluster_master_ip "/usr/local/scripts/SevOne-fix-mysql-permissions"

3. Verify the pair (PAS/HSA) can connect to the Cluster Leader without being rejected.

   on PAS

   $ ssh root@<PAS appliance>

   PAS$ mysql -h $cluster_master_ip -u root -p
   on HSA

   $ ssh root@<HSA appliance>

   HSA$ mysql -h $cluster_master_ip -u root -p

Verify Virtual IP (no Static Routes)

The Virtual IP must be up only on the active appliance. Ensure that the Virtual IP on the passive appliance is not up. Execute the following commands.

Verify Virtual IP (Static Routes configured)

The Virtual IP must be up only on the active appliance. Ensure that the Virtual IP on the passive appliance is not up. Execute the following commands.

using NMCLI

Execute the following steps to change the IP address manually using NMCLI.

1. Using ssh , log into SevOne NMS appliance as root .

   $ ssh root@<NMS appliance>

2. Execute the following command to view all the interfaces.

   $ nmcli

    

   ens160: connected to ens160

   "VMware VMXNET3"

   ethernet (vmxnet3), 00:50:56:BE:C7:12, hw, mtu 1500

   ip4 default

   inet4 10.129.27.33/22

   route4 0.0.0.0/0

   route4 10.129.24.0/22

   inet6 fe80::250:56ff:febe:c712/64

   route6 fe80::/64

   route6 ff00::/8

    

   docker0: unmanaged

   "docker0"

   bridge, 02:42:A9:B2:4F:F5, sw, mtu 1500

    

    

   lo: unmanaged

   "lo"

   loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536

    

    

   DNS configuration:

   servers: 10.168.0.50 10.168.16.50 10.205.8.50

   domains: wifi.sevone.com wilm.sevone.com sevone.com network.qa

   interface: ens160

3. To modify an interface, execute the following command.

   $ nmcli connection modify <interface name> ipv4.address <ip_address/prefix>

   Example

   $ nmcli connection modify ens160 ipv4.address 10.129.27.55/22

   The default gateway, default via, must be in the subnet of the new IP address. However, if the default gateway also needs to be modified, execute following command.

   $ nmcli connection modify ens160 ipv4.gateway <Gateway_IP>

   
   

   Example

   $ nmcli connection modify ens160 ipv4.gateway 10.129.24.0

   
   

using NMTUI

Execute the following steps to change the IP address manually using NMTUI.

1. Using ssh , log into SevOne NMS appliance as root .

   $ ssh root@<NMS appliance>

2. To modify an interface, execute the following command.

   $ nmtui edit <interface name>

   Example

   $ nmtui edit ens160

   Navigate to the IP address to modify

   

Verify IP Address Change at Network-Level

Change IP Address in NMS Configuration

Depending on the NMS appliance on which the network IP address change has been performed, the NMS configuration for that NMS appliance must be updated to reflect the change. Based on the appliance type (primary / secondary) and current role (active / passive), use one of the applicable options below.

Reboot Appliance

If the IP address on the appliance has changed, please reboot the appliance for the new IP address to take effect.

Network Address

When a customer does not want to expose internal IP addresses to the external network, NAT is applied to the internal IP addresses when routing to the external network. There is no need to use a NAT'd IP for Internal Network communications.

NAT IP addresses are routable between External Network and Internal Network only. The hosts in the internal network must use the physical internal IP address to route to other hosts in the internal network.

Only static NAT (one-to-one NAT) is supported. Dynamic NAT/PAT are not supported.

As an example, the basic network addressing scheme is summarized as shown in the table below.

SevOne NMS Peers

SevOne NMS peer functions as a Cluster Leader, a polling PAS or a DNC. A peer typically comprises of two appliances - a primary and a secondary appliance. The standalone peers have only the primary appliance. The primary and secondary appliances may be located in separate subnets or separate physical Data Centers. However, both appliances in a peer reside either in the internal network or the external network. A peer is never split between the internal network and the external network.

Each appliance in the network must have its own unique physical IP address.

SevOne NMS Peers Table

In SevOne NMS, the peers table maintains one record for each peer. Each peer can have only one IP address assigned to its primary and one IP address to its secondary (if a secondary exists). The peer will always be referred by the IP address of the current active appliance for the peer and there can optionally be one Virtual IP assigned to each peer.

The peers table is replicated to all peers and is identical on all peers in the cluster.

For SevOne NMS to function properly, apart from hub-and-spoke clusters, all peers must be reachable in a fully meshed configuration using the peering addresses contained in the peers table.

In standard configuration, SevOne NMS is unable to support the network architecture in which peers in the internal network are addressed using their physical addresses from other peers within the internal network and NAT addresses from peers in the external network.

firewalld

firewalld is a firewall management tool for Linux operating systems which supports the processing of IP packet filtering rules, including NAT, on the Linux host.

By applying firewalld NAT rules, except for hub-and-spoke clusters, it is possible to ensure a full-mesh connectivity between SevOne NMS peers while supporting SevOne NMS clustering and the common peers table addressing in the normal way without any changes to the SevOne NMS application.

Apply firewalld NAT to Internal Network Peers

In this model, the NMS peering address maintained in the peers table is the NAT IP address of the internal peers and the physical IP address of the external peers.

firewalld NAT rules are applied to the internal peers to enable them to communicate with each other using the NAT IP addresses within the internal network.

External peers do not require any firewalld NAT configuration as they can route to the internal peers using the NAT IP addresses stored in the NMS peers table.

Apply firewalld NAT to External Network

In this model, the NMS peering address is the physical IP address of all peers in both internal and external networks.

firewalld NAT rules are applied to each of the external peers to enable them to communicate with the internal peers using the physical IP addresses of internal peers.

Internal peers do not require any firewalld NAT rules configuration since they can route to the external network and the internal peers using their physical addresses.

Create nat.csv & hosts.ini files

1. Using a text editor of your choice, create a nat.csv text file in any directory. For example, /etc/SevOne/nat. The nat.csv file must contain the physical and NAT IP addresses for all internal appliances (where NAT is applied). The format of the .csv file is as follows.

   <physical IP 1>,<NAT IP 1>

   <physical IP 2>,<NAT IP 2>

   ...

   The nat.csv file does not contain the header row.

   Example

   10.168.180.21, 10.133.72.26

   10.168.180.22, 10.133.72.27

   
   

2. Using a text editor of your choice, create a hosts.ini text file in any directory. For example, /etc/SevOne/nat. The hosts.ini file must contain the host IP addresses where NAT rules are applied. The format of the .csv file is as follows.

   <hostip1>

   <hostip2>

   ...

   The hosts.ini file does not contain the header row.

   Example

   10.133.72.21

   10.133.72.22

Generate firewalld NAT Rules

1. Execute the following command to generate NAT rules on all peers mentioned in the hosts.ini file.

   $ SevOne-act firewall apply-nat --natfile /etc/SevOne/nat/nat.csv \

   --ipfile /etc/SevOne/nat/hosts.ini

   The script detects the IP address of the local host and determines the appropriate firewalld NAT rules depending on whether the rules are being applied to a NAT’d host (Internal Network) or a non-NAT’d host (External Network).

2. If the rules are not as expected, correct the nat.csv file. To verify if the NAT rules are correct, please execute the command in Verify Rules.

   1. Execute the following command to flush the rules.

      $ SevOne-act firewall flush-nat --ipfile /etc/SevOne/nat/hosts.ini

   2. Apply NAT rules again.

      $ SevOne-act firewall apply-nat --natfile /etc/SevOne/nat/nat.csv \

      --ipfile /etc/SevOne/nat/hosts.ini

3. To verify if rules are successfully added, execute the following command on each host to check the firewalld NAT rules.

   $ firewall-cmd --direct --get-all-rules

   Check if rules are permanently set

   $ firewall-cmd --permanent --direct --get-all-rules

   You may check /etc/firewalld/direct.xml if rules are added permanently or not.

4. Restart docker if required on each peer in the cluster.

   $ systemctl restart docker

firewall-cmd Commands

The following are useful firewall-cmd commands.

Manual Peer Connectivity Checks

Prior to adding a new peer to the NMS cluster, manual connectivity checks must be performed to ensure that the new peer has connectivity to/from all other peers using the IP address which will be used to add the peer to the NMS cluster.

1. Confirm that each existing peer in the cluster is reachable using ssh and mysql from the new peer using the peer IP address which is contained in the NMS peers table.

   $ ssh <existing peering address>

    

   $ mysql -h <existing peering address>

   MySQL connections will be rejected until the new peer has been added to the NMS cluster. Confirm that the IP address in a connection message is the peering address of the peer attempting to make the connection - it can be either the NAT IP address or the physical IP address which is used for adding a new peer.

2. Confirm that the new peer is reachable using ssh from each existing peer using the peering address of the new peer.

   $ ssh <new peering address>

    

   $ mysql -h <new peering address>

   MySQL connections will be rejected until the new peer has been added to the NMS cluster. Confirm that the IP address in a connection message is the peering address of the peer attempting the connection.

NMS Cluster Connectivity Checks

After the NMS cluster has been built, the standard NMS cluster health check scripts can be used to verify that the required connectivity is working between all peers and to troubleshoot any issues observed during the clustering activity.

1. Execute the following command to perform a full checkout of the cluster.

   $ SevOne-act check checkout –full-cluster

   Individual check scripts can be executed if any errors are detected.

2. To detect issues with connectivity and port access, execute the following command.

   $ SevOne-act check peers --full-cluster -v

    

   $ SevOne-act check mysql-full-mesh-connectivity --full-cluster -v

firewalld Service Checks

1. Execute the following command to check if firewalld service is enabled and active.

   $ firewalld.service - firewalld - dynamic firewall daemon

    

   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)

   Active: active (running) since Tue 2020-06-23 10:00:54 UTC; 1 day 20h ago

   Docs: man:firewalld(1)

   Main PID: 22923 (firewalld)

   Tasks: 2

   Memory: 29.0M

   CGroup: /system.slice/firewalld.service

   └─22923 /usr/bin/python2 -Es /usr/sbin/firewalld --nofork --nopid

What happens if the IP address of NAT'd server is changed?

If the IP address is changed using SevOne-change-ip or if the IP address of the NAT'd appliance is changed, firewalld rules must be regenerated.

What is the procedure to apply the NAT rules if an appliance has to be rebuilt that already has a NAT configuration?

firewalld rules must be regenerated.

What if I add a new peer to the cluster?

If you add a new peer to the cluster, firewalld NAT configuration rules must be applied before the peer is added to the NMS cluster.

What if I need to remove a peer using NAT?

For the peer that has been removed, NAT rules must be flushed out manually. Once removed, you must execute the flush rules script, as mentioned in Generate firewalld NAT Rules, on it. On the remaining peers, NAT rules will have to be applied again with the updated config.

What if I want to change the current Cluster Leader peer where NAT may be configured?

If there is no change in the IP address of NAT’d servers (i.e., current Cluster Leader peer), the NAT rules do not need to be reapplied. However, if there is a change of IP address in any of the peers, the config will have to be flushed and reconfigured by using the NAT utility commands such as SevOne-act firewall flush-nat and SevOne-act firewall apply-nat. Also, described in section Generate firewalld NAT Rules.

What happens if the hostname is changed?

No changes are required.

If NAT is configured, does it require firewalld service to be running?

Yes, firewalld service must always be up and running if NAT is configured.

How about bonding in CentOS when you have active-backup setup?

NAT configuration is not supported over Virtual IP. If bonding is configured to use Virtual IP, it will not be supported.

What steps do I need to follow if I am using a custom port and do not have firewalld enabled?

Please refer to the following sections in SevOne NMS System Administration Guide to add the port rule as required.

• Administration > Cluster Manager > Cluster Settings tab > Firewall subtab.

• Administration > Cluster Manager > [specific appliance] > Peer Settings tab > Firewall subtab.