Difference between revisions of "Junos Evo"

From W9CR
Jump to navigation Jump to search
(Created page with "Page on Juniper OS Evolved Basically juniper laid an egg with this.")
 
Line 1: Line 1:
−
Page on Juniper OS Evolved
+
Page on Juniper OS Evolved (Junos EVO)
  
−
Basically juniper laid an egg with this.
+
Basically juniper laid an egg with this os, and there's very little testing. I think AI would be better.
 +
 
 +
I'm using this on the QFX5130-48c, which is slightly different than the 5130-32CD, but is still Trident Based.
 +
 
 +
 
 +
= Unable to filter number of MACs =
 +
 
 +
QFX5130 can't limit MACs learned on a port.
 +
 
 +
I had to build a script that monitors this and shuts down a port if a limit is exceeded.  [https://github.com/W9CR/qfx5130-mac_limit Link to Github]
 +
 
 +
What's even more fail is the QFX has a mac learning log, but it's not stored on disk or put into syslog.  This means you can't chain an event from it, so the script has to just poll in a loop.  This is just fail.
 +
 
 +
<pre>
 +
bryan@TAMSFLDE-QFX9> show ethernet-switching mac-learning-log
 +
Wed Jul 29 11:07:56 vlan_name TEST-VLAN+2240 mac 0c:c4:7a:d2:ed:c7 was learned on et-0/0/46.2240 with flags: 0x2001f
 +
Wed Jul 29 11:08:14 vlan_name TEST-VLAN+2240 mac 00:00:00:00:01:80 was added to vtep-58.32769 with flags: 0x2413f
 +
Wed Jul 29 11:08:14 vlan_name TEST-VLAN+2240 mac 0c:c4:7a:28:b3:74 was learned on vtep-58.32769 with flags: 0x2001f
 +
Wed Jul 29 11:13:08 vlan_name TEST-VLAN+2240 mac 00:00:00:00:01:80 was deleted with flags: 0x51a0
 +
Wed Jul 29 11:13:08 vlan_name TEST-VLAN+2240 mac 0c:c4:7a:28:b3:74 was deleted with flags: 0x1080
 +
Wed Jul 29 11:13:13 vlan_name TEST-VLAN+2240 mac 00:00:00:00:01:80 was added to vtep-58.32769 with flags: 0x2413f
 +
Wed Jul 29 11:13:13 vlan_name TEST-VLAN+2240 mac 0c:c4:7a:28:b3:74 was learned on vtep-58.32769 with flags: 0x2001f
 +
Wed Jul 29 11:16:55 vlan_name TEST-VLAN+2240 mac 00:00:00:00:01:80 was deleted with flags: 0x51a0
 +
</pre>
 +
 
 +
= Unable to filter per port at L2 =
 +
 
 +
In the service provider interface configuration putting an interface into a mac-vrf EVPN context, there's no way to apply a firewall filter on a per interface basis.  Since the config doesn't have 'family ethernet-switching' there's no way to apply a filter per interface.
 +
 
 +
In this example config, unit 2240 is untagged, as the port's native vlan is 2240.  This also isn't illustrated well in Juniper docs.
 +
<pre>
 +
bryan@TAMSFLDE-QFX9> show configuration interfaces et-0/0/46
 +
description "Fink xg1";
 +
flexible-vlan-tagging;
 +
native-vlan-id 2240;
 +
speed 10g;
 +
mtu 9004;
 +
encapsulation flexible-ethernet-services;
 +
unit 100 {
 +
    vlan-id 100;
 +
    family inet {
 +
        address 10.0.0.1/30;
 +
    }
 +
}
 +
unit 2240 {
 +
    description TEST-LAN;
 +
    encapsulation vlan-bridge;
 +
    vlan-id 2240;
 +
}
 +
</pre>
 +
 
 +
What does work is filtering on the input to the service vlan.  This works as each interface in the service is filtered into before being bridged in the service.
 +
 
 +
<pre>
 +
bryan@TAMSFLDE-QFX9# show routing-instances MGMT-SVC| display inheritance no-comments
 +
/* instance is MAC-VRF, this is default for QFX and supports vlans*/
 +
instance-type mac-vrf;
 +
protocols {
 +
    evpn {
 +
        encapsulation vxlan;
 +
        default-gateway advertise;
 +
        /* allow all vxlan vni's defined under vlans in this service */
 +
        extended-vni-list all;
 +
    }
 +
}
 +
description MGMT-SVC-EVPN;
 +
/* source interface is lo0.0 and ipv6 */
 +
vtep-source-interface lo0.0 inet6;
 +
switch-options {
 +
    mac-ip-table-size {
 +
        5000;
 +
    }
 +
}
 +
/* needs to be vlan-aware, just default and has a per vlan bridging table */
 +
service-type vlan-aware;
 +
/* RD here is the router ID with the ID as defined */
 +
route-distinguisher 149.112.184.226:2200;
 +
/* note that interfaces shouldn't be defined here, only at the VLAN level */
 +
/* the base interfaces should always be in the TBAX EVPN instance */
 +
vrf-target target:10475:2200;
 +
/* define vlans in this */
 +
vlans {
 +
    MGMT-VLAN {
 +
        vlan-id 2200;
 +
        l3-interface irb.2200;
 +
        forwarding-options {
 +
            /* Filter here is needed to prevent stuff across the Service*/
 +
            /* note you can ignore this not found error, it's here, just EVO things! */
 +
            filter {
 +
                input MGMT-LAN-FILTER; ## reference 'MGMT-LAN-FILTER' not found
 +
            }
 +
        }
 +
        switch-options {
 +
            /* set the mac-ip table to 10x what is expected /*
 +
            mac-ip-table-size {
 +
            5000;
 +
            }
 +
            mac-move-limit {
 +
            2;
 +
            packet-action shutdown;
 +
            }
 +
            /* enable mac-stats */
 +
            mac-statistics;
 +
        }
 +
        /* define the VNI's in this, and they must be unique */
 +
        vxlan {
 +
            vni 692200;
 +
        }
 +
    }
 +
}
 +
</pre>
 +
 
 +
= VXLAN encapsulation doesn't work =
 +
In prior than 24.x Junos the '[https://www.juniper.net/documentation/us/en/software/junos/multicast-l2/topics/topic-map/layer-2-forwarding-tables.html PFE UFT Profiles]' Tunnels still worked in the LPM profile just not at scale.  Since this is only a few tunnels, it worked.  At some point they "fixed" this without telling anyone.  There is no notice in the release notes about this.  What's worse is the router give no notice when configuring tunnel protocols with lpm-profile enabled.  Looking at BGP, routeing tables and forwarding tables, it looks correct, even the evpn proxy arp works.  Looking at the interface between routers everything is there; other than VXLAN.
 +
 
 +
The only notice is a pfe error in the logs as below.
 +
<pre>
 +
Jul 27 14:12:24  TAMSFLDE-QFX7 evo-pfemand[1256]: [t:1256] [Error] Nexthop: Failed to creae underlay Nh = 5007in Nh = 9671
 +
Jul 27 14:12:24  TAMSFLDE-QFX7 evo-pfemand[1256]: RAISE_OBJECT_ERROR: Object with guid=579822173966 from app=l2ald-agent on node=re0 objtype=Nexthop reported with error=unknown
 +
Jul 27 14:12:24  TAMSFLDE-QFX7 evo-pfemand[1256]: [t:1256] [Error] Nexthop:  NhAck: Send Error TPA: nhId:9671, type: 19 error: unknown
 +
Jul 27 14:12:24  TAMSFLDE-QFX7 evo-pfemand[1256]: [t:1256] [Error] Nexthop: Failed to creae underlay Nh = 5007in Nh = 9671
 +
</pre>
 +
 
 +
Juniper doesn't publish this on their site for the QFX5130-48c.  It's significantly different than the 32-CD version.
 +
<pre>
 +
bryan@TAMSFLDE-QFX7> show pfe uft-profile-info
 +
SENT: Ukern command: show evo-pfemand uft profile-info
 +
==============================================================================
 +
                  PFE UFT Profiles
 +
==============================================================================
 +
                default-profile  lpm-profile  host-profile  host-acl-profile
 +
==============================================================================
 +
IPV4-host      32K              32K          98K          98K
 +
IPV4-lpm        340K            725K        32K          32K
 +
IPV6-host      16K              16K          49K          49K
 +
IPV6-lpm        174K            386K        18K          18K
 +
L2-mac          32K              32K          96K          96K
 +
FP-compression  20K              0            0            20K
 +
ARP-overlay    32K              64K          32K          32K
 +
ARP-underlay    32K              0            32K          32K
 +
L3-mcast v4    16K              16K          30K          48K
 +
L3-mcast v6    8K              8K          16K          24K
 +
Tunnels        Supported        No support  Supported    Supported
 +
==============================================================================
 +
</pre>
 +
 
 +
Enabling another profile will resolve this issue. 
 +
 
 +
= Software verification PFE crash =
 +
 
 +
 
 +
I did 'request system software rollback' which stages and verifies a rollback.  This caused the PFE to reboot, and all sessions to bounce.  This shouldn't happen.
 +
 
 +
 
 +
<pre>
 +
bryan@TAMSFLDE-QFX7> request system software rollback Read from remote host 23.149.104.25: Operation timed out
 +
Connection to 23.149.104.25 closed.
 +
client_loop: send disconnect: Broken pipe
 +
</pre>
 +
 
 +
'''WTF Juniper; do better.'''
 +
 
 +
<pre>
 +
MKUltra:~ bryan$ ssh qfx7
 +
Last login: Tue Jul 28 12:35:23 2026 from 108.191.164.198
 +
--- JUNOS 25.4R1.13-EVO Linux (none) 5.15.164-10.22.33.18-yocto-standard-juniper-17155-g5adff1d6d404 #1 SMP PREEMPT Tue Oct 28
 +
bryan@TAMSFLDE-QFX7> request system software rollback Starting software rollback to default rollback version
 +
Software rollback in progress...
 +
reø: Validating current config for rollback version junos-evo-install-qfx-ms-x86-64-25.2R1-S2.3-EVO
 +
reø: Validation passed for version junos-evo-install-qfx-ms-x86-64-25.2R1-S2.3-EVO
 +
reø: Copying current config to rollback version
 +
reø: Boot version is now 'junos-evo-install-qfx-ms-x86-64-25.2R1-S2.3-EV0'
 +
reD: Rollback Done. Next Boot version is now "junos-evo-install-qfx-ms-x86-64-25.2R1-S2.3-EVO'. Must reboot now to activate.
 +
reø: Syncing nodes
 +
re0: All nodes synced
 +
Software rollback succeeded.
 +
NOTICE :
 +
'pending' rollback version will be activated at next reboot...
 +
WARNING: Configuration changes made after this point will not be validated.
 +
WARNING: Target image will be booted with the latest configuration. Changing configuration now may be risky.
 +
bryan@TAMSFLDE-QFX7> request system reboot
 +
The entire system (all nodes) will reboot causing traffic loss, do you wish to continue? [yes, nol (no) yes
 +
*** System
 +
shutdown message from root@TAMSFLDE-QFX7 ***
 +
</pre>

Revision as of 05:24, 30 July 2026

Page on Juniper OS Evolved (Junos EVO)

Basically juniper laid an egg with this os, and there's very little testing. I think AI would be better.

I'm using this on the QFX5130-48c, which is slightly different than the 5130-32CD, but is still Trident Based.


Unable to filter number of MACs

QFX5130 can't limit MACs learned on a port.

I had to build a script that monitors this and shuts down a port if a limit is exceeded. Link to Github

What's even more fail is the QFX has a mac learning log, but it's not stored on disk or put into syslog. This means you can't chain an event from it, so the script has to just poll in a loop. This is just fail. 

bryan@TAMSFLDE-QFX9> show ethernet-switching mac-learning-log
Wed Jul 29 11:07:56 vlan_name TEST-VLAN+2240 mac 0c:c4:7a:d2:ed:c7 was learned on et-0/0/46.2240 with flags: 0x2001f
Wed Jul 29 11:08:14 vlan_name TEST-VLAN+2240 mac 00:00:00:00:01:80 was added to vtep-58.32769 with flags: 0x2413f
Wed Jul 29 11:08:14 vlan_name TEST-VLAN+2240 mac 0c:c4:7a:28:b3:74 was learned on vtep-58.32769 with flags: 0x2001f
Wed Jul 29 11:13:08 vlan_name TEST-VLAN+2240 mac 00:00:00:00:01:80 was deleted with flags: 0x51a0
Wed Jul 29 11:13:08 vlan_name TEST-VLAN+2240 mac 0c:c4:7a:28:b3:74 was deleted with flags: 0x1080
Wed Jul 29 11:13:13 vlan_name TEST-VLAN+2240 mac 00:00:00:00:01:80 was added to vtep-58.32769 with flags: 0x2413f
Wed Jul 29 11:13:13 vlan_name TEST-VLAN+2240 mac 0c:c4:7a:28:b3:74 was learned on vtep-58.32769 with flags: 0x2001f
Wed Jul 29 11:16:55 vlan_name TEST-VLAN+2240 mac 00:00:00:00:01:80 was deleted with flags: 0x51a0

Unable to filter per port at L2

In the service provider interface configuration putting an interface into a mac-vrf EVPN context, there's no way to apply a firewall filter on a per interface basis. Since the config doesn't have 'family ethernet-switching' there's no way to apply a filter per interface.

In this example config, unit 2240 is untagged, as the port's native vlan is 2240. This also isn't illustrated well in Juniper docs. 

bryan@TAMSFLDE-QFX9> show configuration interfaces et-0/0/46
description "Fink xg1";
flexible-vlan-tagging;
native-vlan-id 2240;
speed 10g;
mtu 9004;
encapsulation flexible-ethernet-services;
unit 100 {
    vlan-id 100;
    family inet {
        address 10.0.0.1/30;
    }
}
unit 2240 {
    description TEST-LAN;
    encapsulation vlan-bridge;
    vlan-id 2240;
}

What does work is filtering on the input to the service vlan. This works as each interface in the service is filtered into before being bridged in the service. 

bryan@TAMSFLDE-QFX9# show routing-instances MGMT-SVC| display inheritance no-comments
/* instance is MAC-VRF, this is default for QFX and supports vlans*/
instance-type mac-vrf;
protocols {
    evpn {
        encapsulation vxlan;
        default-gateway advertise;
        /* allow all vxlan vni's defined under vlans in this service */
        extended-vni-list all;
    }
}
description MGMT-SVC-EVPN;
/* source interface is lo0.0 and ipv6 */
vtep-source-interface lo0.0 inet6;
switch-options {
    mac-ip-table-size {
        5000;
    }
}
/* needs to be vlan-aware, just default and has a per vlan bridging table */
service-type vlan-aware;
/* RD here is the router ID with the ID as defined */
route-distinguisher 149.112.184.226:2200;
/* note that interfaces shouldn't be defined here, only at the VLAN level */
/* the base interfaces should always be in the TBAX EVPN instance */
vrf-target target:10475:2200;
/* define vlans in this */
vlans {
    MGMT-VLAN {
        vlan-id 2200;
        l3-interface irb.2200;
        forwarding-options {
            /* Filter here is needed to prevent stuff across the Service*/
            /* note you can ignore this not found error, it's here, just EVO things! */
            filter {
                input MGMT-LAN-FILTER; ## reference 'MGMT-LAN-FILTER' not found
            }
        }
        switch-options {
            /* set the mac-ip table to 10x what is expected /*
            mac-ip-table-size {
            5000;
            }
            mac-move-limit {
            2;
            packet-action shutdown;
            }
            /* enable mac-stats */
            mac-statistics;
        }
        /* define the VNI's in this, and they must be unique */
        vxlan {
            vni 692200;
        }
    }
}

VXLAN encapsulation doesn't work

In prior than 24.x Junos the 'PFE UFT Profiles' Tunnels still worked in the LPM profile just not at scale. Since this is only a few tunnels, it worked. At some point they "fixed" this without telling anyone. There is no notice in the release notes about this. What's worse is the router give no notice when configuring tunnel protocols with lpm-profile enabled. Looking at BGP, routeing tables and forwarding tables, it looks correct, even the evpn proxy arp works. Looking at the interface between routers everything is there; other than VXLAN.

The only notice is a pfe error in the logs as below. 

Jul 27 14:12:24  TAMSFLDE-QFX7 evo-pfemand[1256]: [t:1256] [Error] Nexthop: Failed to creae underlay Nh = 5007in Nh = 9671
Jul 27 14:12:24  TAMSFLDE-QFX7 evo-pfemand[1256]: RAISE_OBJECT_ERROR: Object with guid=579822173966 from app=l2ald-agent on node=re0 objtype=Nexthop reported with error=unknown
Jul 27 14:12:24  TAMSFLDE-QFX7 evo-pfemand[1256]: [t:1256] [Error] Nexthop:  NhAck: Send Error TPA: nhId:9671, type: 19 error: unknown
Jul 27 14:12:24  TAMSFLDE-QFX7 evo-pfemand[1256]: [t:1256] [Error] Nexthop: Failed to creae underlay Nh = 5007in Nh = 9671

Juniper doesn't publish this on their site for the QFX5130-48c. It's significantly different than the 32-CD version. 

bryan@TAMSFLDE-QFX7> show pfe uft-profile-info
SENT: Ukern command: show evo-pfemand uft profile-info
==============================================================================
                   PFE UFT Profiles
==============================================================================
                default-profile  lpm-profile  host-profile  host-acl-profile
==============================================================================
IPV4-host       32K              32K          98K           98K
IPV4-lpm        340K             725K         32K           32K
IPV6-host       16K              16K          49K           49K
IPV6-lpm        174K             386K         18K           18K
L2-mac          32K              32K          96K           96K
FP-compression  20K              0            0             20K
ARP-overlay     32K              64K          32K           32K
ARP-underlay    32K              0            32K           32K
L3-mcast v4     16K              16K          30K           48K
L3-mcast v6     8K               8K           16K           24K
Tunnels         Supported        No support   Supported     Supported
==============================================================================

Enabling another profile will resolve this issue.

Software verification PFE crash

I did 'request system software rollback' which stages and verifies a rollback. This caused the PFE to reboot, and all sessions to bounce. This shouldn't happen. 


bryan@TAMSFLDE-QFX7> request system software rollback Read from remote host 23.149.104.25: Operation timed out
Connection to 23.149.104.25 closed.
client_loop: send disconnect: Broken pipe

WTF Juniper; do better. 

MKUltra:~ bryan$ ssh qfx7
Last login: Tue Jul 28 12:35:23 2026 from 108.191.164.198
--- JUNOS 25.4R1.13-EVO Linux (none) 5.15.164-10.22.33.18-yocto-standard-juniper-17155-g5adff1d6d404 #1 SMP PREEMPT Tue Oct 28
bryan@TAMSFLDE-QFX7> request system software rollback Starting software rollback to default rollback version
Software rollback in progress...
reø: Validating current config for rollback version junos-evo-install-qfx-ms-x86-64-25.2R1-S2.3-EVO
reø: Validation passed for version junos-evo-install-qfx-ms-x86-64-25.2R1-S2.3-EVO
reø: Copying current config to rollback version
reø: Boot version is now 'junos-evo-install-qfx-ms-x86-64-25.2R1-S2.3-EV0'
reD: Rollback Done. Next Boot version is now "junos-evo-install-qfx-ms-x86-64-25.2R1-S2.3-EVO'. Must reboot now to activate.
reø: Syncing nodes
re0: All nodes synced
Software rollback succeeded.
NOTICE :
'pending' rollback version will be activated at next reboot...
WARNING: Configuration changes made after this point will not be validated.
WARNING: Target image will be booted with the latest configuration. Changing configuration now may be risky.
bryan@TAMSFLDE-QFX7> request system reboot
The entire system (all nodes) will reboot causing traffic loss, do you wish to continue? [yes, nol (no) yes
*** System
shutdown message from root@TAMSFLDE-QFX7 ***
Retrieved from "https://wiki.w9cr.net/index.php?title=Junos_Evo&oldid=8501"