Compiled xt_TPROXY again, for Kong’s r39960M buildI just upgraded my router to ddwrt r39960 (Kong’s build), and found that xt_TPROXY.ko no longer works.
Since kernel ABI is not stable, it’s not too much a surprise, and I tried rebuilding xt_TPROXY.ko. Given that I’ve done it before, I didn’t expect it to be too much troublesome.
And it was.
xt_TPROXY.ko compiled from dd-wrt source code r39960 (mirror) using the same steps before failed to load, complaining:
1 insmod: cannot insert 'xt_TPROXY.ko': unknown symbol in module
dmesg further showed the offending symbols are related to RCU:
12 [ 1000.000000] xt_TPROXY: Unknown symbol __rcu_read_lock (err 0)[ 1000.000000] xt_TPROXY: Unknown symbol __rcu_read_unlock (err 0)
nm confirmed this:
1234 $ nm xt_TPROXY.ko U __rcu_read_lock U __rcu_read_unlock...
But the same (undefined) symbols did not appear in the old xt_TPROXY.ko.
include/linux/rcupdate.h (simplified below) showed that these symbols were compiled into undefined symbols only if CONFIG_PREEMPT_RCU was defined:
1234567891011121314151617181920 #ifdef CONFIG_PREEMPT_RCU void __rcu_read_lock(void);void __rcu_read_unlock(void); #else /* #ifdef CONFIG_PREEMPT_RCU */ static inline void __rcu_read_lock(void){ if (IS_ENABLED(CONFIG_PREEMPT_COUNT)) preempt_disable();} static inline void __rcu_read_unlock(void){ if (IS_ENABLED(CONFIG_PREEMPT_COUNT)) preempt_enable();} #endif /* #else #ifdef CONFIG_PREEMPT_RCU */
Before r39661, dd-wrt (more precisely, .config_northstar_smp) used TREE_CPU, in which case, __rcu_read_lock and __rcu_read_unlock were inlined and wouldn’t introduce new undefined symbols. That’s why the old xt_TPROXY.ko did not suffer from this.
This is also where things get very weird. Since Kong’s new builds are based on r39960, it should have also been configured with CONFIG_PREEMPT_RCU. However, uname -v disagreed:
12 # uname -v#666 SMP Sat Jun 8 11:15:19 CEST 2019
(Note that PREEMPT does not show in uname -v‘s output.)
So I suspect either Kong used his own .config file, or he somehow forgot to svn up his workspace and used a older .config_northstar_smp (for example, r39661). The former looks more likely, as r39661’s .config_northstar_smp does not match Kong’s firmware very precisely.
Anyway, I used r39661’s .config_northstar_smp to replace r39960’s, and on top of it, enabled CONFIG_NETFILTER_XT_TARGET_TPROXY to compile xt_TPROXY.ko.
And it worked.
To make things a little easier, this time I didn’t use dd-wrt’s own toolchain, instead, I used Linaro’s toolchain, which is much smaller (~274M vs ~4G), and enough for building kernel modules. ARM’s toolchain should probably work too, but I didn’t test it.
Note that you also do not have to download the whole dd-wrt source code, only dd-wrt/src/linux/universal/linux-4.4 (in my case, Linux 4.4 should be used. YMMV.) is enough.
I still had to remove several missing (proprietary?) Kconfig from drivers/net/wireless/Kconfig this time, as they were missing from dd-wrt’s (public) source tree.
After modifying .config and patching drivers/net/wireless/Kconfig, the following commands should work well:
12345 # Add toolchain/bin to PATH before running these commands.make ARCH=arm CROSS_COMPILE=arm-eabi- oldconfigmake ARCH=arm CROSS_COMPILE=arm-eabi- preparemake ARCH=arm CROSS_COMPILE=arm-eabi- modules_preparemake ARCH=arm CROSS_COMPILE=arm-eabi- modules SUBDIRS=net/netfilter
I still used my workaround for xt_TPROXY for IPv6 in Linux 4.4. I didn’t test xt_TPROXY.ko without the patch, but linux 4.4.180’s source code seemed not to contain the official fix for this issue so I nontheless patched xt_TPROXY.c before compiling it.
I just upgraded my router to ddwrt r39960 (Kong’s build), and found that xt_TPROXY.ko no longer works.
Since kernel ABI is not stable, it’s not too much a surprise, and I tried rebuilding xt_TPROXY.ko. Given that I’ve done it before, I didn’t expect it to be too much troublesome.
And it was.
xt_TPROXY.ko compiled from dd-wrt source code r39960 (mirror) using the same steps before failed to load, complaining:
1 | insmod: cannot insert 'xt_TPROXY.ko': unknown symbol in module |
dmesg further showed the offending symbols are related to RCU:
1 2 | [ 1000.000000] xt_TPROXY: Unknown symbol __rcu_read_lock (err 0) [ 1000.000000] xt_TPROXY: Unknown symbol __rcu_read_unlock (err 0) |
nm confirmed this:
1 2 3 4 | $ nm xt_TPROXY.ko U __rcu_read_lock U __rcu_read_unlock ... |
But the same (undefined) symbols did not appear in the old xt_TPROXY.ko.
include/linux/rcupdate.h (simplified below) showed that these symbols were compiled into undefined symbols only if CONFIG_PREEMPT_RCU was defined:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | #ifdef CONFIG_PREEMPT_RCU void __rcu_read_lock(void); void __rcu_read_unlock(void); #else /* #ifdef CONFIG_PREEMPT_RCU */ static inline void __rcu_read_lock(void) { if (IS_ENABLED(CONFIG_PREEMPT_COUNT)) preempt_disable(); } static inline void __rcu_read_unlock(void) { if (IS_ENABLED(CONFIG_PREEMPT_COUNT)) preempt_enable(); } #endif /* #else #ifdef CONFIG_PREEMPT_RCU */ |
Before r39661, dd-wrt (more precisely, .config_northstar_smp) used TREE_CPU, in which case, __rcu_read_lock and __rcu_read_unlock were inlined and wouldn’t introduce new undefined symbols. That’s why the old xt_TPROXY.ko did not suffer from this.
This is also where things get very weird. Since Kong’s new builds are based on r39960, it should have also been configured with CONFIG_PREEMPT_RCU. However, uname -v disagreed:
1 2 | # uname -v #666 SMP Sat Jun 8 11:15:19 CEST 2019 |
(Note that PREEMPT does not show in uname -v‘s output.)
So I suspect either Kong used his own .config file, or he somehow forgot to svn up his workspace and used a older .config_northstar_smp (for example, r39661). The former looks more likely, as r39661’s .config_northstar_smp does not match Kong’s firmware very precisely.
Anyway, I used r39661’s .config_northstar_smp to replace r39960’s, and on top of it, enabled CONFIG_NETFILTER_XT_TARGET_TPROXY to compile xt_TPROXY.ko.
And it worked.
To make things a little easier, this time I didn’t use dd-wrt’s own toolchain, instead, I used Linaro’s toolchain, which is much smaller (~274M vs ~4G), and enough for building kernel modules. ARM’s toolchain should probably work too, but I didn’t test it.
Note that you also do not have to download the whole dd-wrt source code, only dd-wrt/src/linux/universal/linux-4.4 (in my case, Linux 4.4 should be used. YMMV.) is enough.
I still had to remove several missing (proprietary?) Kconfig from drivers/net/wireless/Kconfig this time, as they were missing from dd-wrt’s (public) source tree.
After modifying .config and patching drivers/net/wireless/Kconfig, the following commands should work well:
1 2 3 4 5 | # Add toolchain/bin to PATH before running these commands. make ARCH=arm CROSS_COMPILE=arm-eabi- oldconfig make ARCH=arm CROSS_COMPILE=arm-eabi- prepare make ARCH=arm CROSS_COMPILE=arm-eabi- modules_prepare make ARCH=arm CROSS_COMPILE=arm-eabi- modules SUBDIRS=net/netfilter |
I still used my workaround for xt_TPROXY for IPv6 in Linux 4.4. I didn’t test xt_TPROXY.ko without the patch, but linux 4.4.180’s source code seemed not to contain the official fix for this issue so I nontheless patched xt_TPROXY.c before compiling it.