Present Location: News >> Blog


> BIRD Musings
Posted by prox, from Seattle, on February 22, 2017 at 00:09 local (server) time

I started messing with BIRD the other day to work around some IPv6 issues with Quagga.  The configuration is fairly simple, but I ran into a weird issue where it picks the wrong interface IPv4 address from some of my OpenVPN tunnels.  For example, I've got these interfaces:

(storm:0:06:EST)% ip a s tun0
5: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1456 qdisc pfifo_fast state UNKNOWN group default qlen 100
    inet peer scope global tun0
       valid_lft forever preferred_lft forever
(storm:0:06:EST)% ip a s tun1
4: tun1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1456 qdisc pfifo_fast state UNKNOWN group default qlen 100
    inet peer scope global tun1
       valid_lft forever preferred_lft forever

Here's what BIRD sees:

bird> show route protocol direct1       dev lo [direct1 23:53:06] * (240)     dev tun0 [direct1 23:53:06] * (240)     dev tun1 [direct1 23:53:06] * (240)   dev eth0 [direct1 23:53:06] * (240)

The addresses shown are the remote end of the OpenVPN tunnels, not the local end, which I'd expect.


Update: Well, this should have been obvious:

(storm:0:34:EST)% ip r s p kernel   dev tun0 scope link src dev tun1 scope link src dev eth0 scope link src

I suppose I'll have to figure out how to get the link source to be visible to BIRD so I can advertise it, which is one of my odd requirements here.

Comments: 0
> apt-get SIGABRT
Posted by prox, from Seattle, on February 05, 2017 at 22:10 local (server) time

As I was doing upgrades on a few of my Raspberry Pi machines (running Raspbian), I ran into a situation where apt-get would SIGABRT on any action.  Depending on the shell, the message to the user is a little different and also overwrites a different amount of APT's status message.  Here's Zsh:

(storm:20:50:EST)% sudo apt-get --purge remove quagga-pimd quagga-isisd quagga-ospfd quagga-ospf6d
zsh: abort      sudo apt-get --purge remove quagga-pimd quagga-isisd quagga-ospfd

Here's Bash:

prox@storm:~$ sudo apt-get --purge remove quagga-pimd 
Aborted package lists... 48%

No matter what the operation (install, upgrade, update, remove), APT would just get a SIGABRT.  There was nothing in the kernel log buffer and the RPi 3 I was on at the time wasn't out of disk space or memory.  Since this RPi had just completed a dist-upgrade and I had rebooted it, everything was more-or-less up-to-date with the "testing" channel of Raspbian.  I searched the web and couldn't find anything related to "abort" or "SIGABRT" on Debian or Raspbian.  There were also no bugs filed against any of the relevant packages.  So, I figured that I had just hit a bug and started downgrading.  I used dpkg-repack on another RPi 3 of the following packages, which I downgraded to:


On the upgraded RPi, I'd been running APT 1.4~beta4, libc 2.24-9, and dpkg 1.18.18.  Unfortunately, none of the downgrades did the trick.  My next step was to downgrade the kernel, but I started to think something was maybe gummed up with APT itself, possibly in its database since it always received the SIGABRT about 50% of the way through reading package lists.  I ended up looking up the various storage directories and decided to clear out /var/lib/apt/lists.  I did another apt-get upgrade and things started working again:

(storm:21:55:EST)% sudo apt-get update
Get:1 jessie InRelease [22.9 kB]
Get:2 testing InRelease [15.0 kB]
Get:3 jessie/main armhf Packages [141 kB]                   
Get:4 testing/main armhf Packages [11.7 MB]           
Get:5 jessie/ui armhf Packages [53.6 kB]                
Get:6 testing/contrib armhf Packages [56.0 kB]                                                                     
Get:7 testing/non-free armhf Packages [95.1 kB]                                                                    
Get:8 testing/rpi armhf Packages [1,360 B]                                                                         
Fetched 12.1 MB in 24s (493 kB/s)                                                                                                                              
Reading package lists... Done
(storm:21:59:EST)% sudo apt-get --purge remove quagga-pimd quagga-isisd quagga-ospfd quagga-ospf6d
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libquagga0 quagga-bgpd quagga-core quagga-ripd quagga-ripngd
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  quagga* quagga-isisd* quagga-ospf6d* quagga-ospfd* quagga-pimd*
0 upgraded, 0 newly installed, 5 to remove and 8 not upgraded.
After this operation, 957 kB disk space will be freed.
Do you want to continue? [Y/n] 

Apparently something had gotten corrupted in the downloaded package lists and APT didn't handle it properly.  I'd submit a bug for this but I didn't think to backup /var/lib/apt/lists beforehand so I'm not sure how to reproduce this.  Anyway, if you get a SIGABRT when running apt-get, rm /var/lib/apt/lists/* and things will be happy again.

Also, in other news, Lady Gaga's Super Bowl LI performance was pretty decent.  The drones were a nice touch.

Comments: 0
> Updates & C-States Silliness on Linux
Posted by prox, from Seattle, on October 11, 2016 at 02:25 local (server) time

I haven't blogged in awhile.. so here are some updates.

I recently moved dax, which is the FreeBSD-based host that powers this web server, to a VM and away from Internap's Agile service.  dax used to be a dedicated server hosted by Voxel dot net and provided IPv6 connectivity to the rest of my network since September of 2005.  However, things started going downhill when Internap acquired them in 2012.  Technical support turned into a horror story and they recently indicated to me (in an IPv6 support ticket from hell) that the Agile "legacy" services were going to be done away with later in 2016 (not sure if this is true).

I migrated all of my sites off the IPv6 PA /56 I had from Internap and onto my own PI /44.  I moved dax to a VM on excalibur, a dedicated server hosted by Choopa, LLC.  I'm now Internap-free and running everything off of AS395460.

Anyway, excalibur's got an Intel Xeon E3-1240 v2 CPU with 4x cores and should run at 3.40 GHz.  Normally, with servers of mine that run VMs, I instruct cpufreq-set(1) to run all cores with the performance governor, which is supposed to direct all cores to run at their maximum clock rate.  However, with the E3 Xeon CPUs, this doesn't work.  The clock frequencies of each oscillate based on load and cause a considerable latency penalty, which is fairly visible in packet forwarding (causes jitter).  Here's a snapshot:

(excalibur:2:12:EDT)% cat /proc/cpuinfo|grep -i MHz
cpu MHz		: 1714.742
cpu MHz		: 2652.000
cpu MHz		: 3563.492
cpu MHz		: 3799.898
cpu MHz		: 3722.070
cpu MHz		: 2126.195
cpu MHz		: 3800.164
cpu MHz		: 2313.062

Searching on the web indicated I would need to completely disable the C-states in the BIOS, something I wasn't really willing to do and didn't feel like the correct solution.  I then came across another post that indicated I should write "0" to /dev/cpu_dma_latency but keep the file open.  So, I did this:

(excalibur:2:12:EDT)# cat > /dev/cpu_dma_latency

.. and didn't hit ^D.  Sure enough:

(excalibur:2:12:EDT)% cat /proc/cpuinfo|grep -i MHz
cpu MHz		: 3599.882
cpu MHz		: 3599.882
cpu MHz		: 3599.882
cpu MHz		: 3599.882
cpu MHz		: 3599.882
cpu MHz		: 3599.882
cpu MHz		: 3599.882
cpu MHz		: 3599.882

Seriously?  There's a whole nice structure in /sys/devices/system/cpu/cpufreq/* that Linux has used for over a decade to control CPU frequency scaling and we've now a one-off /dev character device that controls such things on a modern CPU?

Well, considering the stupidity of systemd that's supposedly accepted by most Linux distributions now, I guess I shouldn't be surprised that this type of hacky interface exists.  At least I've got a way to emulate the behavior of the performance governor without mucking with BIOS settings.  It turns out this also works on my E3 1245 v3 Xeon I've at home in vega, too.

Update: This is a bad idea.  This causes all CPUs to run very hot even when utilization is low, which confuses me:

excalibur burning

Update: The right solution to this problem is to just disable the Intel P-states driver by passing intel_pstate=disable on the kernel command line.  The acpi-cpufreq driver is used instead and operates the way it should.  See comments for more information.

Comments: 10
> Linux VLAN Interface Renaming
Posted by prox, from Seattle, on July 10, 2016 at 01:38 local (server) time

Almost as a continuation of Dead RPis and Interface Naming, I've now encountered some systemd/udev-related problems when creating VLAN interfaces on Linux (Debian) on one of my routers after upgrading to bleeding-edge stuff (Linux 4.6.0 and udev/systemd 230):

Say I have one of the following in /etc/network/interfaces:

auto eth1.35
iface eth1.35 inet static


auto vlan35
iface vlan35 inet static
     vlan-raw-device eth1

I'll get this:

(starfire:22:23:PDT)# ifup vlan35
Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
Added VLAN with VID == 35 to IF -:eth1:-
(starfire:22:24:PDT)# ip link show dev vlan35
Device "vlan35" does not exist.
(starfire:22:24:PDT)# ip link|grep rename
61: rename61@eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

Obviously, I expected vlan35 to be created, but udev renamed it immediately to rename61.  The only udev rules that are acting on this, from what I can tell, are /etc/udev/rules.d/76-netnames.rules and /lib/udev/rules.d/19-ifrename.rules:

(starfire:22:27:PDT)# cat /lib/udev/rules.d/19-ifrename.rules
# udev rules to properly integrate ifrename.
# Renaming is done using /etc/iftab, with full ifrename functionality.
# Require udev version 107 or later.
# Please double check the path to ifrename, and make sure its available
# when udev runs (i.e. on boot partition).

# Enable this rule to test with udevtest.
#ENV{UDEV_LOG}=="6", SUBSYSTEM=="net", ACTION=="add", IMPORT{program}="/sbin/ifrename -D -V -u -i %k", NAME:="$env{INTERFACE}"

# Main ifrename rule.
# If interface is found in /etc/iftab, subsequent rename rules are bypassed.
# If interface is not found in /etc/iftab, subsequent rename rules applies.
SUBSYSTEM=="net", ACTION=="add", IMPORT{program}="/sbin/ifrename -u -i %k", NAME:="$env{INTERFACE}"
(starfire:22:27:PDT)# cat /etc/udev/rules.d/76-netnames.rules 
# Generated by on Sat Jul  9 19:09:59 PDT 2016
SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:1b:21:08:2b:0a", NAME="eth0"
SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:04:23:5f:4c:d9", NAME="eth1"
SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:1b:21:08:2b:0b", NAME="eth2"
SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:04:23:5f:4c:d8", NAME="eth3"
SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:08:74:c5:d4:61", NAME="eth4"

The former comes with the udev package but the latter was created by me since I needed all of the NICs in my router to have consistent and traditional names.  I'd add these VLAN interfaces to 76-netnames.rules but there's nothing unique I can match since the MAC addresses on the VLAN interfaces are copied from the physical interface.

It's pretty obvious that udev is renaming VLAN interfaces and getting in the way.  I should normally be able to hack around this by a pre-up or up directive in /etc/network/interfaces but the renameXX interfaces are not consistent (but predictable:

(starfire:22:31:PDT)# ip link show |grep rename
(starfire:22:31:PDT)# vconfig add eth1 99      
Added VLAN with VID == 99 to IF -:eth1:-
(starfire:22:31:PDT)# ip link show |grep rename
63: rename63@eth1:  mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
(starfire:22:31:PDT)# vconfig rem rename63     
Removed VLAN -:rename63:-
(starfire:22:31:PDT)# ip link show |grep rename
(starfire:22:31:PDT)# vconfig add eth1 99      
Added VLAN with VID == 99 to IF -:eth1:-
(starfire:22:31:PDT)# ip link show |grep rename
64: rename64@eth1:  mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000

I suspect adding net.ifnames=0 to the kernel command line will fix things again but that's not a long-term solution.  Right now I'm left with no option but to build these interfaces manually after boot.  I'll update this post when I log a bug against one of the Debian packages involved here.

Update 20160710

I managed to get this to work by sticking the following in my /etc/udev/rules.d/76-netnames.rules:

SUBSYSTEM=="net", KERNEL=="eth*.*" ACTION=="add", NAME="$kernel"
SUBSYSTEM=="net", KERNEL=="vlan*" ACTION=="add", NAME="$kernel"

This causes udev to not touch VLAN interfaces using the naming schemes I care about (DEV_PLUS_VID_NO_PAD and VLAN_PLUS_VID_NO_PAD).

Comments: 0
> Dead RPis and Interface Naming
Posted by prox, from Seattle, on May 22, 2016 at 22:18 local (server) time

I recently had a case where a few of my [very] remote Raspberry Pis dropped off the network upon reboot (power failure, etc.).  I run the 'testing' release of Raspbian & Debian on my Linux boxes so I do an apt-get dist-upgrade rather often.  The RPis that dropped off the network had indeed been upgraded recently so I had one shipped back to me for investigation.

It turns out that eth0 and wlan0 had been renamed to enx$MAC and looked like this:

Previously wlan0

(I had the RPi plugged into my TV)

I do know about predictable network interface names, which is supposed to prevent interface names changing when hardware is swapped, among other things.  The enx$MAC naming scheme is used for USB devices.  Up until now I've been content to disable this feature by adding net.ifnames=0 to the kernel command-line.  Indeed, I had this flag in /boot/cmdline.txt on the Pis but it just isn't working anymore.

It turns out that this feature is set to be deprecated in Debian 10, along with the automatically-generated /etc/udev/rules.d/70-persistent-net.rules file, which is appended when a new NIC is added to the system.  This is documented in /usr/share/doc/udev/README.Debian.gz, apparently.

Unfortunately for me, the RPis never had 70-persistent-net.rules written and apparently udev 229-5 already ignores net.ifnames=0.  One of my dist-upgrades bumped udev from 229-4 to 229-5.  This is why none of the interface names that were defined in /etc/network/interfaces were found on boot and the RPis booted up with no enabled interfaces.

/usr/share/doc/udev/README.Debian.gz goes on to say that if custom interface names are needed, use 76-netnames.rules with the following content, which looks similar to what was previously written out in 70-persistent-net.rules automatically:

SUBSYSTEM=="net", ACTION=="add", ATTR{address}=="00:50:56:1a:aa:ab", NAME="eth0"

There's no mention of this method of interface naming being deprecated so I'm going to use it for now.

This is just the latest annoyance, for me, when it comes to Linux's interface naming.  Several years back the right way to setup predictable interface names was to use /etc/iftab and ifrename(8)!

What I haven't figured out is how this is going to impact VMs and virtio.  My own VMs with VirtualBox have static MAC addresses assigned to the vbox file so using 76-netnames.rules will work fine.  However, I'm not sure if EC2 and other Xen-based VPS providers always provide a constant MAC address.  One of my EC2 instances has 53-ec2-network-interfaces.rules, which handles some sort of hotplugging.  I'm not sure if that script will stop working as well after some time and udev upgrades but it would seem the solution is to use some ATTR other than 'addresss'.

Comments: 0
> Watch out for Quagga upgrades!
Posted by prox, from Seattle, on May 07, 2016 at 14:00 local (server) time

I upgraded dax from FreeBSD 10.2 to FreeBSD 10.3 this morning.  Since I haven't upgraded ports in awhile I upgraded net/quagga from to 1.0.20160315 and boy that was a mistake!

First off, my IPv6 default route wasn't getting redistributed anymore.  I've got a static IPv6 route in zebra pointing to Internap's upstream router:

ipv6 route ::/0 2001:48c8:1:2::1

It's seen fine by zebra:

S>* ::/0 [1/0] via 2001:48c8:1:2::1, em0

However, bgpd doesn't see it at all:

dax_bgpd# show ipv6 bgp ::/0
% Network not in table

The redistribution statement I've got in bgpd isn't doing any good, apparently:

router bgp 65304
 bgp router-id
 address-family ipv6
 redistribute static route-map IPv6_ADVERTISE_STATICS
ipv6 prefix-list IPv6_DEFAULT seq 5 permit ::/0
route-map IPv6_ADVERTISE_STATICS permit 10
 match ipv6 address prefix-list IPv6_DEFAULT

I managed to get this working without restarting the daemons by doing this a few times in zebra:

no ipv6 route ::/0 2001:48c8:1:2::1
ipv6 route ::/0 2001:48c8:1:2::1

This fixed it once, but it wasn't consistent, so that's a fail.

While I was trying to fix this, I noticed that some of my single-homed VPN tunnels from dax appeared down, but it was because bgpd wasn't consistently advertising my IPv6 prefixes over the EBGP sessions.  I couldn't figure it out but I didn't spend too much time on it.  2001:48c8:1:1ff::3a is an example VPN neighbor that isn't getting all of the IPv6 routes and 2001:48c8:1:105::/64 is a LAN prefix in Seattle, where my desktop sits.

(dax:13:31:EDT)% show ipv6 bgp 2001:48c8:1:105::/64
BGP routing table entry for 2001:48c8:1:105::/64
Paths: (4 available, best #4, table Default-IP-Routing-Table)
  Advertised to non peer-group peers:
  2001:48c8:1:1ff::6 2001:48c8:1:1ff::16 2001:48c8:1:1ff::1e 2001:48c8:1:1ff::32 2001:48c8:1:1ff::3a 2001:48c8:1:1ff::52 2001:48c8:1:1ff::56

bgpd says it's advertising it above just fine to 2001:48c8:1:1ff::3a, but advertising-routes says otherwise.  The remote box, firefly, doesn't see it at all.  I lost the command output when I closed the terminal in a fit of rage.

Anyway, I downgraded to using ports-mgmt/portdowngrade and rebuilt.  All of the problems magically disappeared.  I suppose I'll try to reproduce this on a FreeBSD VM and see what I can figure with some debugging enabled.

I'm thinking of switching to BIRD but I'll have to do it when I've got a bit more time on my hands.

Comments: 0
> Zsh + MPlayer
Posted by prox, from Seattle, on April 12, 2016 at 23:27 local (server) time

From the "tiny blogs" department...

Yep, I'm a Zsh fan.  I've always been one and probably won't change.

One of the nifty features of Zsh is the ability to expand wildcards on the command-line in a particular order.  For me, this often comes in handy when combined with some sort of media player.  For example, what if I wanted to watch videos in a directory with MPlayer and start with the most recently modified ones?  Zsh has that covered:

% mplayer *.mp4(oa)

The (oa) tells Zsh to expand '*' to all files in $CWD but to order the expansion by last modified.  There are a few other sort specifiers, but I find 'a' the most useful.  Using a capital 'O' will reverse the sort order.

Comments: 0
> OpenSSH 7.x and Keys
Posted by prox, from York, on December 26, 2015 at 13:22 local (server) time

OpenSSH 7.0 was released a few months ago and deprecated both the ssh-dss and ssh-rsa keys used for SSHv2 public key authentication.  I haven't found a definitive source stating why these key types were deprecated other than some issue with entropy (doesn't make sense to me because that sounds like a machine-specific problem).  I, unfortunately, still use these keys quite a bit and it's not feasible to completely convert to one of the newer key types.

OpenSSH 7.0 appeared in FreeBSD ports pretty quickly and recently made its way into Debian testing (stretch).

So, what are the newer key types supported?  From what I can tell, it's just ecdsa and ed25519 for SSHv2.  From the OpenSSH 6.2 ssh-keygen(1) manpage:

     -t type
             Specifies the type of key to create.  The possible values are
             ``rsa1'' for protocol version 1 and ``dsa'', ``ecdsa'' or ``rsa''
             for protocol version 2.

From the OpenSSH 7.1 ssh-keygen(1) manpage:

     -t dsa | ecdsa | ed25519 | rsa | rsa1
             Specifies the type of key to create.  The possible values are
             "rsa1" for protocol version 1 and "dsa", "ecdsa", "ed25519", or
             "rsa" for protocol version 2.

Even though dsa and rsa keys are still listed above in the 7.1 man page as being capable of created, they're not accepted by default anymore by ssh or sshd:

debug1: Next authentication method: publickey
debug1: Skipping ssh-dss key /home/prox/.ssh/id_dsa for not in PubkeyAcceptedKeyTypes

(if you're wondering, and I was too, DSS is a document that describes the creation of DSA keys as answered here)

Running ssh -Q key will also dump the list of keys acceptable by ssh:

(nox:11:46:CST)% ssh -Q key

The solution seems obvious, just throw out the old keys and use an ecdsa key, right?  Sure, that'll work for OpenSSH versions that support it.  However, sometimes I have to log into a few legacy boxes that only support RSA and DSA keys (Solaris, IRIX, random network devices, etc. - I've got some old stuff!).

What about keeping around the old keys?  Sure, we can use PubkeyAcceptedKeyTypes in sshd_config and ssh_config like this:

PubkeyAcceptedKeyTypes ssh-dss,ssh-rsa

The only problem is that this option only exists in 7.0 and above.  I use a common ~/.ssh/ssh_config for all of my systems and OpenSSH 6.x barfs on that line.

What's the solution?  Well, one is to not upgrade to OpenSSH 7.0, but that's just delaying the inevitable.  My solution may just be to use two keys, one for modern systems and one for very old systems that don't support ecdsa or ed25519.  Regardless, it's pretty annoying, but security always is, right?

Update 20151229: This page highlights some of these differences and workarounds, too.

Comments: 0

No Previous PageDisplaying page 1 of 119 of 947 results Next Page