I just upgraded my router to ddwrt r39960 (Kong’s build), and found that xt_TPROXY.ko no longer works.

Since kernel ABI is not stable, it’s not too much a surprise, and I tried rebuilding xt_TPROXY.ko. Given that I’ve done it before, I didn’t expect it to be too much troublesome.

And it was.

xt_TPROXY.ko compiled from dd-wrt source code r39960 (mirror) using the same steps before failed to load, complaining:

insmod: cannot insert 'xt_TPROXY.ko': unknown symbol in module

dmesg further showed the offending symbols are related to RCU:

[ 1000.000000] xt_TPROXY: Unknown symbol __rcu_read_lock (err 0)
[ 1000.000000] xt_TPROXY: Unknown symbol __rcu_read_unlock (err 0)

nm confirmed this:

$ nm xt_TPROXY.ko
         U __rcu_read_lock
         U __rcu_read_unlock
...

But the same (undefined) symbols did not appear in the old xt_TPROXY.ko.

include/linux/rcupdate.h (simplified below) showed that these symbols were compiled into undefined symbols only if CONFIG_PREEMPT_RCU was defined:

#ifdef CONFIG_PREEMPT_RCU

void __rcu_read_lock(void);
void __rcu_read_unlock(void);

#else /* #ifdef CONFIG_PREEMPT_RCU */

static inline void __rcu_read_lock(void)
{
    if (IS_ENABLED(CONFIG_PREEMPT_COUNT))
        preempt_disable();
}

static inline void __rcu_read_unlock(void)
{
    if (IS_ENABLED(CONFIG_PREEMPT_COUNT))
        preempt_enable();
}

#endif /* #else #ifdef CONFIG_PREEMPT_RCU */

Before r39661, dd-wrt (more precisely, .config_northstar_smp) used TREE_CPU, in which case, __rcu_read_lock and __rcu_read_unlock were inlined and wouldn’t introduce new undefined symbols. That’s why the old xt_TPROXY.ko did not suffer from this.

This is also where things get very weird. Since Kong’s new builds are based on r39960, it should have also been configured with CONFIG_PREEMPT_RCU. However, uname -v disagreed:

# uname -v
#666 SMP Sat Jun 8 11:15:19 CEST 2019

(Note that PREEMPT does not show in uname -v‘s output.)

So I suspect either Kong used his own .config file, or he somehow forgot to svn up his workspace and used a older .config_northstar_smp (for example, r39661). The former looks more likely, as r39661’s .config_northstar_smp does not match Kong’s firmware very precisely.

Anyway, I used r39661’s .config_northstar_smp to replace r39960’s, and on top of it, enabled CONFIG_NETFILTER_XT_TARGET_TPROXY to compile xt_TPROXY.ko.

And it worked.

To make things a little easier, this time I didn’t use dd-wrt’s own toolchain, instead, I used Linaro’s toolchain, which is much smaller (~274M vs ~4G), and enough for building kernel modules. ARM’s toolchain should probably work too, but I didn’t test it.

Note that you also do not have to download the whole dd-wrt source code, only dd-wrt/src/linux/universal/linux-4.4 (in my case, Linux 4.4 should be used. YMMV.) is enough.

I still had to remove several missing (proprietary?) Kconfig from drivers/net/wireless/Kconfig this time, as they were missing from dd-wrt’s (public) source tree.

After modifying .config and patching drivers/net/wireless/Kconfig, the following commands should work well:

# Add toolchain/bin to PATH before running these commands.
make ARCH=arm CROSS_COMPILE=arm-eabi- oldconfig
make ARCH=arm CROSS_COMPILE=arm-eabi- prepare
make ARCH=arm CROSS_COMPILE=arm-eabi- modules_prepare
make ARCH=arm CROSS_COMPILE=arm-eabi- modules SUBDIRS=net/netfilter

I still used my workaround for xt_TPROXY for IPv6 in Linux 4.4. I didn’t test xt_TPROXY.ko without the patch, but linux 4.4.180’s source code seemed not to contain the official fix for this issue so I nontheless patched xt_TPROXY.c before compiling it.