FritzBox 7170 29.04.57-freetz-devel-2234 crashes with kernel: dst cache overflow

[jhaprins]

Hi everyone,

For some time now on a regular basis I get the following error in my logfile coming from my 7170:

May 30 20:25:13 192.168.178.1 kernel: dst cache overflow
May 30 20:32:42 192.168.178.1 kernel: dst cache overflow
May 30 20:32:42 192.168.178.1 kernel: dst cache overflow
May 30 20:36:42 192.168.178.1 kernel: dst cache overflow
May 30 20:36:42 192.168.178.1 kernel: dst cache overflow

When this happens I better start planning a reboot for my router because otherwise it will crash completly within 30 minutes after the first message. You can understand that most of the time I'm too late and I notice it when all my internet traffic comes to a halt.

I have been looking for a reason for this to happen and what I noticed is that when this happens the ip_dst_cache in the kernel has reached it's limit of 2048 and then it's done with the router. To me it looks like there is som kind of ip_dst_cache leak in the kernel and I've been looking through kernel changes related to this and the only one that I found related to this same message is a patch in the 2.6.12 kernel to fix this and after checking the source tree for the fritzbox kernel, it looks like this patch is in it.

You can see the size of this cache by doing:
cat /proc/slabinfo | grep ip_dst_cache

Are there more people that notice this problem, or am I the only one?

Greetings,
Jan Hugo Prins

 

[Silent-Tears]

My Box is up for a forthnight now, and nothing of that had happend.

 

[jhaprins]

You say your box is up for 14 days now.
But this long isn't a problem for my box either.
The problem is that after some time the box crashes with the said error.
Could you give the output of this command on your box:

cat /proc/slabinfo | grep ip_dst_cache

Jan Hugo

 

[jhaprins]

Today my box needed a reboot again:

/var/mod/root # grep ip_dst_cache /proc/slabinfo
ip_dst_cache 8192 8205 256 15 1 : tunables 120 60 0 : slabdata 547 547 0

He has been working for 16 days.
The last reboot was may 30th, about the same time.

Jan Hugo

 

[Izzy]

I noticed my box to hang from time to time. Just this morning I came to my desk: The LEDs looked fine, but no network. Couldn't even ping the box. Luckily I collect the syslog on a remote server - and guess what I found there?

Dec  3 23:19:52 fritz.box kernel: dst cache overflow
Dec  3 23:20:57 fritz.box kernel: dst cache overflow
Dec  3 23:20:57 fritz.box last message repeated 3 times
... [ a couple of more times ]
Dec  4 02:43:40 fritz.box kernel: dst cache overflow
Dec  4 02:45:49 fritz.box kernel: kdsld: internet: set_snd_ipaddr: 79.207.223.147
Dec  4 02:45:49 fritz.box kernel: kdsld: internet: connected
Dec  4 02:45:49 fritz.box kernel: kdsld: PPP led: on (value=1)
Dec  4 02:45:53 192.168.101.1 kernel: printk: 65 messages suppressed.
Dec  4 02:45:53 192.168.101.1 kernel: dst cache overflow
Dec  4 02:45:54 192.168.101.1 multid[568]: 0.0.0.0:2097: connect to 63.208.196.94:80 failed - No buffer space available (132)

After that, there are a couple of more "dst cache overflow" entries - but at this point the network is dead. Shortly after this, I find the entries from SIP not being available (voipd[597]: [email protected]: REGISTER failed 2 status 500 (try again in 10 seconds)) - which is one of the indications.

Though I doubt that is Freetz specific (I have those hangs from time to time as far as I can think back, even before I used Freetz), I'm of course interested in a solution, too

Best regards,
Izzy.

 

[Izzy]

I just found an interesting ticket at OpenWRT, describing exactly the same problem, same results, same everything: See here. They obviously also found the source to it, i.e. where to fix the issue - it has something to do with MAC filtering. Maybe one of the devs can check whether there is something to be done by Freetz - or to delegate it to AVM? At OpenWRT, they got it fixed (they changed something at the bridge firewall modules, according to the revision mentioned - but I cannot see what that should be).

Besides: Google lists 3250 results for '+kernel +"dst cache overflow"'. All I checked seem to point to some bug in the kernel...

 

[hermann72pb]

By the way: In OpenWRT ticket contrack module of IPTABLES is named in connection with this overflow problem. I am not expert in IPTABLES, but only as idea: Is it possible, that this overflow problem causes contrack crash?

Best regards,

 

[olistudent]

Attached you can find a utiltiy to monitor the rt_cache. I'm awaiting your results. ;-)

Regards
Oliver

 

[Nexium]

"dst cache overflow" problem workaround

Hi,
i'm using a 7170 with 29.04.59freetz-devel-2826 and I've the very same problem.
As a workaround I've increased value in /proc/sys/net/ipv4/route/max_size to 32768 and now the system is stable (uptime 10 days).
The ip_dst_cache is still growing but it's under the limit (now is 17000).

Hope it helps.

Cheers,
Max

 

[olistudent]

Hi.
Is it growing over time? As I understand there should be a garbage collector which should run automatically and clean up old entries.

Regards
Oliver

 

[jhaprins]

dst cache overflow

Yes, till now here it has been growing over time after every reboot.
Last Saturday I installed the latest version on my FB7170 and rebooted several times so it is now very low again.
But I will keep an eye on it and tell you what it does here.

The tool you send earlier in a message, should I put that on my FB and run it?
If so, where should I put it? And should I run it on regular intervals ? What does it do?

Jan Hugo

--
FB7170: 7170_04.67freetz-devel-2907

 

[olistudent]

Hi.
Yes. Please put it on the box. Do you have an USB Stick attached? Perhaps you can put the log there. Perhaps it's enough if you run it every couple of days and write down the results.
It shows the dst cache entries.

Regards
Oliver

 

[jhaprins]

Hi

I have the tool running but in the output there is nothing related to the ip_dst_cache as far as I can see.
I have also started a script on the box that greps the ip_dst_cache line out op /proc/slabinfo and puts this in a file.
In a few days we have more info.

Jan Hugo Prins

 

[olistudent]

Perhaps I compiled the wrong tool. :mrgreen:
But as I remember I tested it...

Regards
Oliver

 

[jhaprins]

Some results.

I have created some log files of the ip_dst_cache and the lnstat info.
It looks as if the ip_dst_cache isn't exploding anymore since I have upgraded to 29.04.67.
But this could also be because I'm not at home a lot at the moment, so there is not very much traffic.
I will test more the coming weeks.
Is my assumption right that the ip_dst_cache is somewhat the same as the rt_cache ??

Greetings,
Jan Hugo Prins.

 

[olistudent]

Hi.
Everything looks fine in this log. The garbage collector seems to work. How long was the period you logged?

Regards
Oliver

 

[jhaprins]

The lines are 5 seconds apart.
I'm gonna create a new logfile with lines that are 5 minutes apart and I will let this run for some time.
But for now, it looks like my problem is solved in the new firmware.

Jan Hugo