Freetz für AVM DVB-C Repeater verfügbar?

[PeterPawn]

Any strides meanwhile?

 

[DM41]

@baribal I merged several posts of yours in this thread. Please edit your last post if you just want to add something. Thanks.

 

[koyaanisqatsi]

And i extend...
And do a @baribal ( to PeterPawn) for recognize him that news.
...like you get informed with a red sign in the upper right bell image

 

[baribal]

Sorry, busy at work. I tried your unpack_kernel.sh script on kernel.image and got:

freetz@freetz:~/YourFritz/avm_kernel_config$ ./unpack_kernel.sh <~/freetz-ng/var/tmp/kernel.image
./unpack_kernel.sh: 106: Bad substitution
./unpack_kernel.sh: 107: Bad substitution
./unpack_kernel.sh: 106: Bad substitution
./unpack_kernel.sh: 107: Bad substitution
lzma: (stdin): File format not recognized

I also tried another approach:

freetz@freetz:~/freetz-ng$ ./fwmod -u dl/fw/FRITZ.Box_WLAN_Repeater_DVB_C.en-de-es-it-fr-pl.133.07.01.image

STEP 1: UNPACK
source firmware file: dl/fw/FRITZ.Box_WLAN_Repeater_DVB_C.en-de-es-it-fr-pl.133.07.01.image
checking signature: disabled
unpacking firmware image
removing NMI vector from SquashFS
NMI vector v3 found at offset 0xBE0000, removing it ... done.
splitting kernel image
unpacking filesystem image
    Filesystem on dl/fw/FRITZ.Box_WLAN_Repeater_DVB_C.en-de-es-it-fr-pl.133.07.01.image.mod/original/kernel/kernelsquashfs.raw is xz compressed (4:0)
    Parallel unsquashfs: Using 1 processor
    2429 inodes (2777 blocks) to write
    created 2036 files
    created 220 directories
    created 381 symlinks
    created 12 devices
    created 0 fifos
unpacking var.tar
detected firmware version: DVB-C_en-de-es-fr-it-pl 133.07.01 rev63069 {ALL} (07.11.2018 16:32:58)
done.

FINISHED

Now I have those two files:

-rw-r--r-- 1 freetz freetz  1540608 Aug  9 12:15 kernel.raw
-rw-r--r-- 1 freetz freetz 10841438 Aug  9 12:15 kernelsquashfs.raw

And when I try to build from sources I get the following:

freetz@freetz:~/YourFritz/avm_kernel_config$ make
make: *** No rule to make target 'linux/scripts/dtc/libfdt/fdt_addresses.c', needed by 'linux/scripts/dtc/libfdt/fdt.o'.  Stop.

I have the "linux" dir with sources inside:

freetz@freetz:~/YourFritz/avm_kernel_config$ ll
total 108
drwxr-xr-x  4 freetz freetz  4096 Aug  9 12:35 ./
drwxr-xr-x 28 freetz freetz  4096 Jul 27 20:42 ../
-rw-r--r--  1 freetz freetz     0 Aug  9 12:20 1.1
-rw-r--r--  1 freetz freetz  6128 Jul 27 20:42 avm_kernel_config_helpers.c
-rw-r--r--  1 freetz freetz  1050 Jul 27 20:42 avm_kernel_config_helpers.h
-rw-r--r--  1 freetz freetz  1326 Jul 27 20:42 avm_kernel_config_macros.h
-rwxr-xr-x  1 freetz freetz  6443 Jul 27 20:42 dump_kernel_config.sh*
-rw-r--r--  1 freetz freetz  8810 Jul 27 20:42 extract_avm_kernel_config.c
-rw-r--r--  1 freetz freetz  9733 Jul 27 20:42 gen_avm_kernel_config.c
-rw-r--r--  1 freetz freetz 18046 Jul 27 20:42 LICENSE
drwxr-xr-x 25 freetz freetz  4096 Nov  7  2018 linux/
-rw-r--r--  1 freetz freetz  1769 Jul 27 20:42 Makefile
drwxr-xr-x  2 freetz freetz  4096 Jul 27 20:42 patches/
-rw-r--r--  1 freetz freetz  1166 Jul 27 20:42 README.md
-rwxr-xr-x  1 freetz freetz 14281 Jul 27 20:42 unpack_kernel.sh*

@PeterPawn btw, I tried your other advice, set FREETZ_AVM_PROP_KERNEL_CONFIG_AREA_SIZE_0=n instead of =y inside the .config and built the image again but I still can't find any "avm_kernel_config_area*" files.

 

[PeterPawn]

Sorry, busy at work.

No matter ... as long as you're still interested in a solution, you may continue later.

But please tell us (esp. me), when you'll start your next try or any further attempts.

It helps me to keep this thread/theme on-screen - I haven't enabled any notifications (by e-mail) and so I have to look actively for news.

Addressing an answer explicitly to me (as proposed by @koyaanisqatsi) really helps - even if I still have to open a IPPF page to receive these notifications.

 

[PeterPawn]

The kernel image from this firmware file by AVM: https://ftp.avm.de/fritzwlan/fritzw...her/fritz.os/FRITZ.Repeater_1750E-07.27.image CONTAINS a avm_kernel_config area for sure.

00468000  80 4c 80 10 00 00 00 00  00 00 00 00 00 00 00 00  |.L..............|
00468010  00 00 00 01 80 4c de a8  00 00 00 02 80 4c dd e8  |.....L.......L..|
00468020  00 00 00 05 80 4c be b0  00 00 00 06 80 4c 9f 78  |.....L.......L.x|
00468030  00 00 00 0c 80 4c 80 40  00 00 01 07 00 00 00 00  |.....L.@........|
00468040  d0 0d fe ed 00 00 1f 38  00 00 00 38 00 00 1c 10  |.......8...8....|
00468050  00 00 00 28 00 00 00 11  00 00 00 10 00 00 00 00  |...(............|
00468060  00 00 03 28 00 00 1b d8  00 00 00 00 00 00 00 00  |...(............|

• at 0x468040 the FDT magic value was found
• at the previous 4K boundary the address of a _avm_kernel_config structure was found, each entry contains an index value as tag and a pointer to the structure with this type of data
• the first entry has a tag of 1 (avm_kernel_config_tags_module_memory) and points to a (memory) address of 0x804cdea8
• the difference of 0x804c8000 and 0x468000 is 0x80060000 -> the two high order bits of a MIPS address have to be masked out and the load address of the (unpacked) kernel seems to be 0x60000, so we have to add/substract this value to convert between absolute memory addresses and file offsets in our unpacked file
• the second entry has a tag of 2 (avm_kernel_config_tags_version_info) and points to a _avm_kernel_version_info structure at 0x804cdde8, containing three strings of 32, 32 and 128 byte (incl. the final NUL character)
• to verify this structure, we have to convert the memory address to a file offset: 0x804cdde8 - 0x80060000 = 0x46dde8 -> we have to look at this offset from our kernel file

0046dde0  65 76 69 73 69 6f 6e 00  00 00 00 00 00 00 00 00  |evision.........|
0046ddf0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
0046de20  00 00 00 00 00 00 00 00  30 37 2e 32 37 00 00 00  |........07.27...|
0046de30  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
0046dea0  00 00 00 00 00 00 00 00  80 4c e0 4c 00 01 b6 a0  |.........L.L....|
0046deb0  00 00 81 e0 00 00 1a 68  80 4c e0 50 00 00 3a f8  |.......h.L.P..:.|

At the expected offset starts the structure, the first two strings (64 byte overall) are empty and the structure member firmwarestring contains the string 07.27 - it ends at offset 0x46de28 + 128 = 0x46dea8 ... and here starts obviously the first entry of the _avm_module_memory structure array. So far the data looks as expected ...

• back at offset 0x468020, we see three FDT blobs for three different HWSubrevisions of the 1750E - for Subrevision 0 (tag 5), 1 (tag 6 at 0x468028) and 7 (tag 0x0c = 12 at 0x468030)
• the last entry (tag 107 at offset 0x468038) marks the end of our array
• at 0x468040 (which will be loaded at memory address 0x804c8040) the third embedded FDT (from entry with tag 7 12) starts

All this together proves, that AVM's kernel for the latest 1750E firmware version (07.27) CONTAINS an avm_kernel_config mechanism - the next task is it now, to extract this area and to generate assembler source files from the extracted data - once for the DVB-C model (from an older firmware) and once for the 1750E model. I'm rather sure, even the older version 07.01 (for DVB-C) contains such an area, too ... but I'm unsure, whether AVM changed the structures of other data (not the FDTs) between version 07.0x and 07.2x.

At least I had to change some code from my older avm_kernel_config project to compile it with the current sources for 1750E (07.27) - but those devices are anyhow the models with the newest Linux kernel version (4.4) in AVM's portfolio.

According to the provided source files by AVM, the configuration area has a size of 96 KB:

95         .data : {       /* Data */
96                 . = . + DATAOFFSET;             /* for CONFIG_MAPPED_KERNEL */
97
98                 INIT_TASK_DATA(THREAD_SIZE)
99                 NOSAVE_DATA
100                 CACHELINE_ALIGNED_DATA(1 << CONFIG_MIPS_L1_CACHE_SHIFT)
101                 READ_MOSTLY_DATA(1 << CONFIG_MIPS_L1_CACHE_SHIFT)
102                 DATA_DATA
103                 CONSTRUCTORS
104 #ifdef CONFIG_AVM_KERNEL
105                 . = ALIGN(4 * 1024);
106                 __avm_kernel_config_start = .;
107                 . += 96 * 1024;
108                 __avm_kernel_config_end = .;
109 #endif
110         }

and this makes it possible for us, to copy this area simply using a dd command:

peh@vidar:/tmp/YourFritz> dd if=kernel.unpacked of=1750E_07.27_avm_kernel_config.bin bs=4096 skip=$(( 0x468000 / 4096 )) count=$(( 96 * 1024 / 4096 ))
24+0 records in
24+0 records out
98304 bytes (98 kB, 96 KiB) copied, 0.000351668 s, 280 MB/s
peh@vidar:/tmp/YourFritz> avm_kernel_config/extract_avm_kernel_config -s 96 kernel.unpacked > 1750E_07.27_avm_kernel_config.bin2
peh@vidar:/tmp/YourFritz> cmp -l 1750E_07.27_avm_kernel_config.bin 1750E_07.27_avm_kernel_config.bin2
peh@vidar:/tmp/YourFritz>

The second attempt shows, how to use my utility to extract this area ... and the final comparision shows, that both methods have identical results.

Now we may use the other utility to get assembler sources from the extracted data:

peh@vidar:/tmp/YourFritz> avm_kernel_config/gen_avm_kernel_config 1750E_07.27_avm_kernel_config.bin > avm_kernel_config_area.1750E_07.27.S
peh@vidar:/tmp/YourFritz>

The created file was attached to this post ... with a .txt extension due to limitations by the board software. To repeat those steps yourself, you have to update your YourFritz checkout first. After this refresh, you should be able to compile the sources from avm_kernel_config subfolder (of my YourFritz repository) at any Linux host with a reasonably up to date OS version - but read the README.md file thoroughly. If you get any error related to a missing fdt_addresses.c file, you may apply the attached patch to AVM's sources in folder linux/scripts/dtc/libfdt - there's one more failure in the supplied source archive by AVM (at least I've found this one with 1750E sources for 07.27).

It's now your part to extract this area from a kernel image for the DVB-C model ... please wait with further post until you've finished or an insoluble problem was found.

EDIT: @Mods ... ich füge die Beiträge später selbst zusammen - ich wollte nur den neuen Teil nicht an den vorherigen Beitrag anhängen, um den Adressaten nicht unnötig zu verwirren (der Stoff ist für die meisten schon schwer genug).

 

[PeterPawn]

Here're the "full steps" I've done in front of the findings from #46:

peh@vidar:/tmp> git clone https://github.com/PeterPawn/YourFritz.git
Cloning into 'YourFritz'...
remote: Enumerating objects: 3540, done.
remote: Counting objects: 100% (27/27), done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 3540 (delta 9), reused 18 (delta 6), pack-reused 3513
Receiving objects: 100% (3540/3540), 4.07 MiB | 10.39 MiB/s, done.
Resolving deltas: 100% (2261/2261), done.
peh@vidar:/tmp> cd YourFritz/
peh@vidar:/tmp/YourFritz> wget http://osp.avm.de/fritzwlan/fritzwlan-repeater-1750e/source-files-FRITZ.Repeater_1750E-07.27.tar.gz
--2021-08-09 14:50:30--  http://osp.avm.de/fritzwlan/fritzwlan-repeater-1750e/source-files-FRITZ.Repeater_1750E-07.27.tar.gz
Resolving osp.avm.de (osp.avm.de)... 217.110.143.116, 2001:920:197e:400::116
Connecting to osp.avm.de (osp.avm.de)|217.110.143.116|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 527266314 (503M) [application/octet-stream]
Saving to: 'source-files-FRITZ.Repeater_1750E-07.27.tar.gz'

source-files-FRITZ.Repeater_1750E-07.27.tar.gz              100%[====================================>] 502.84M  13.1MB/s    in 41s

2021-08-09 14:51:11 (12.2 MB/s) - 'source-files-FRITZ.Repeater_1750E-07.27.tar.gz' saved [527266314/527266314]

peh@vidar:/tmp/YourFritz> ls -l source*
-rw-r--r-- 1 peh users 527266314 Jul  8 17:25 source-files-FRITZ.Repeater_1750E-07.27.tar.gz
peh@vidar:/tmp/YourFritz> tar -tvf source-files-FRITZ.Repeater_1750E-07.27.tar.gz
-rw-r--r-- jenkins/user 249845 2021-07-06 16:15 GPL/asec.tar.gz
-rw-r--r-- jenkins/user 347113945 2021-07-06 16:17 GPL/gcc.tar.gz
-rw-r--r-- jenkins/user     13377 2021-07-06 16:17 GPL/GPL-avmacllib2.tar.gz
-rw-r--r-- jenkins/user     46173 2021-07-06 16:17 GPL/GPL-bridge-utils.tar.gz
-rw-r--r-- jenkins/user    745007 2021-07-06 16:17 GPL/GPL-iproute2.tar.gz
-rw-r--r-- jenkins/user     48876 2021-07-06 16:17 GPL/hui.tar.gz
-rw-r--r-- jenkins/user 174716441 2021-07-06 16:17 GPL/kernel.tar.gz
-rw-r--r-- jenkins/user   2758652 2021-07-06 16:17 GPL/LGPL-libexif.tar.gz
-rw-r--r-- jenkins/user     89316 2021-07-06 16:17 GPL/lte_tools.tar.gz
-rw-r--r-- jenkins/user   1516619 2021-07-06 16:18 GPL/target_tools.tar.gz
-rw-r--r-- jenkins/user        45 2021-07-06 16:18 GPL/usb_host_tools.tar.gz
-rw-r--r-- jenkins/user    430122 2021-07-06 16:18 GPL/wlan.tar.gz
peh@vidar:/tmp/YourFritz> tar -xzf source-files-FRITZ.Repeater_1750E-07.27.tar.gz -O GPL/kernel.tar.gz | tar -tz | sed -e "s|^\([^/]*\)/.*\$|\1|" | uniq
FRITZ!OS Kernel
linux
peh@vidar:/tmp/YourFritz> tar -xzf source-files-FRITZ.Repeater_1750E-07.27.tar.gz -O GPL/kernel.tar.gz | tar -xz linux/; du -hsc linux
731M    linux
731M    total
peh@vidar:/tmp/YourFritz> find linux -name "*libfdt*"
linux/arch/arm/boot/compressed/libfdt_env.h
linux/arch/powerpc/boot/libfdt-wrapper.c
linux/arch/powerpc/boot/libfdt_env.h
linux/include/linux/libfdt.h
linux/include/linux/libfdt_env.h
linux/scripts/dtc/libfdt
linux/scripts/dtc/libfdt/Makefile.libfdt
linux/scripts/dtc/libfdt/libfdt.h
linux/scripts/dtc/libfdt/libfdt_env.h
linux/scripts/dtc/libfdt/libfdt_internal.h
peh@vidar:/tmp/YourFritz> cd avm_kernel_config/
peh@vidar:/tmp/YourFritz/avm_kernel_config> cat README.md
# Create a `avm_kernel_config` section for own kernels on FRITZ!OS-based devices

This folder contains some utilities to replace/recreate missing parts from AVM's open-source package.

AVM invented a mechanism to initialize some internal structures at kernel startup from a model-specific part and there seems to be
no way to get these sources from vendor - at least an explicit demand to provide them was more or less ignored. The sources pointed
out in their answer didn't contain any changes regarding these missing parts.

So one of the possible solutions was reverse engineering the content of this area and the files provided in this folder are a first
approach to take this way.

You can find further information in an IPPF thread (but it's in German only):

<http://www.ip-phone-forum.de/showthread.php?t=287995>

If you want to compile the contained sources for a specific model, you have to provide a symlink named "linux" to the root of the
correct kernel sources. The files "include/uapi/linux/avm_kernel_config.h" and the whole directory "scripts/dtc/libfdt" (from the
OpenFirmware device-tree compiler) are the parts needed from current kernel sources.
peh@vidar:/tmp/YourFritz/avm_kernel_config> ln -s ../linux linux
#
# There is a failure in the provided source files ... the Makefile for libfdt refers to a non-existing file fdt_addresses.c - but this isn't ever 
# required for our purposes. So the reference is erased from Makefile with the attached patch file (remove the .txt extension yourself).
#
peh@vidar:/tmp/YourFritz/avm_kernel_config> patch -d linux/scripts/dtc/libfdt -p0 < libfdt_makefile.patch
patching file Makefile.libfdt
peh@vidar:/tmp/YourFritz/avm_kernel_config> make
gcc -static -std=c99 -m32 -ggdb -I./linux/scripts/dtc/libfdt -I. -I./linux/include -D__KERNEL__ -c linux/scripts/dtc/libfdt/fdt.c -o linux/scripts/dtc/libfdt/fdt.o
gcc -static -std=c99 -m32 -ggdb -I./linux/scripts/dtc/libfdt -I. -I./linux/include -D__KERNEL__ -c linux/scripts/dtc/libfdt/fdt_ro.c -o linux/scripts/dtc/libfdt/fdt_ro.o
gcc -static -std=c99 -m32 -ggdb -I./linux/scripts/dtc/libfdt -I. -I./linux/include -D__KERNEL__ -c linux/scripts/dtc/libfdt/fdt_wip.c -o linux/scripts/dtc/libfdt/fdt_wip.o
gcc -static -std=c99 -m32 -ggdb -I./linux/scripts/dtc/libfdt -I. -I./linux/include -D__KERNEL__ -c linux/scripts/dtc/libfdt/fdt_sw.c -o linux/scripts/dtc/libfdt/fdt_sw.o
gcc -static -std=c99 -m32 -ggdb -I./linux/scripts/dtc/libfdt -I. -I./linux/include -D__KERNEL__ -c linux/scripts/dtc/libfdt/fdt_rw.c -o linux/scripts/dtc/libfdt/fdt_rw.o
gcc -static -std=c99 -m32 -ggdb -I./linux/scripts/dtc/libfdt -I. -I./linux/include -D__KERNEL__ -c linux/scripts/dtc/libfdt/fdt_strerror.c -o linux/scripts/dtc/libfdt/fdt_strerror.o
gcc -static -std=c99 -m32 -ggdb -I./linux/scripts/dtc/libfdt -I. -I./linux/include -D__KERNEL__ -c linux/scripts/dtc/libfdt/fdt_empty_tree.c -o linux/scripts/dtc/libfdt/fdt_empty_tree.o
rm linux/scripts/dtc/libfdt/libfdt.a 2>/dev/null || true
ar rcu linux/scripts/dtc/libfdt/libfdt.a linux/scripts/dtc/libfdt/fdt.o linux/scripts/dtc/libfdt/fdt_ro.o linux/scripts/dtc/libfdt/fdt_wip.o linux/scripts/dtc/libfdt/fdt_sw.o linux/scripts/dtc/libfdt/fdt_rw.o linux/scripts/dtc/libfdt/fdt_strerror.o linux/scripts/dtc/libfdt/fdt_empty_tree.o
ranlib linux/scripts/dtc/libfdt/libfdt.a
gcc -static -std=c99 -m32 -ggdb -O2 -W -Wall -I./linux/scripts/dtc/libfdt -I. -I./linux/include -D__KERNEL__ -c avm_kernel_config_helpers.c -o avm_kernel_config_helpers.o
gcc -static -std=c99 -m32 -ggdb -O2 -W -Wall -I./linux/scripts/dtc/libfdt -I. -I./linux/include -D__KERNEL__ -c gen_avm_kernel_config.c -o gen_avm_kernel_config.o
gcc -static -std=c99 -m32 -ggdb -O2 -W -Wall -I./linux/scripts/dtc/libfdt -I. -I./linux/include -D__KERNEL__ -c extract_avm_kernel_config.c -o extract_avm_kernel_config.o
gcc -static -m32 -L. -o gen_avm_kernel_config gen_avm_kernel_config.o avm_kernel_config_helpers.o ./linux/scripts/dtc/libfdt/libfdt.a
gcc -static -m32 -L. -o extract_avm_kernel_config extract_avm_kernel_config.o avm_kernel_config_helpers.o ./linux/scripts/dtc/libfdt/libfdt.a
peh@vidar:/tmp/YourFritz/avm_kernel_config> cd ..
peh@vidar:/tmp/YourFritz> wget https://ftp.avm.de/fritzwlan/fritzwlan-repeater-1750e/other/fritz.os/FRITZ.Repeater_1750E-07.27.image
--2021-08-09 17:05:40--  https://ftp.avm.de/fritzwlan/fritzwlan-repeater-1750e/other/fritz.os/FRITZ.Repeater_1750E-07.27.image
Resolving ftp.avm.de (ftp.avm.de)... 217.110.95.227, 212.42.224.73, 212.42.224.74, ...
Connecting to ftp.avm.de (ftp.avm.de)|217.110.95.227|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 15646720 (15M) [application/octet-stream]
Saving to: 'FRITZ.Repeater_1750E-07.27.image'

FRITZ.Repeater_1750E-07.27.image                            100%[====================================>]  14.92M  12.6MB/s    in 1.2s

2021-08-09 17:05:41 (12.6 MB/s) - 'FRITZ.Repeater_1750E-07.27.image' saved [15646720/15646720]

peh@vidar:/tmp/YourFritz> tar -xf FRITZ.Repeater_1750E-07.27.image -O ./var/tmp/kernel.image > 1750E_07.27.image
peh@vidar:/tmp/YourFritz> grep -abo sqsh 1750E_07.27.image
1565952:sqsh
peh@vidar:/tmp/YourFritz> dd if=1750E_07.27.image of=kernel.image bs=4096 count=$(( 1565952 / 4096 ))
382+0 records in
382+0 records out
1564672 bytes (1.6 MB, 1.5 MiB) copied, 0.00438198 s, 357 MB/s
peh@vidar:/tmp/YourFritz> bash avm_kernel_config/unpack_kernel.sh < kernel.image > kernel.unpacked
lzma: (stdin): Unexpected end of input
#
# A quick check, whether the unpacked file contains strings with at least 16 characters, containing the word "gpio".
# The unexpected end of input above may be ignored, if the kernel was unpacked successfully.
#
peh@vidar:/tmp/YourFritz> strings -n 16 kernel.unpacked | grep gpio | wc -l
182
#
# It's an unpacked kernel for sure - now we can try to extract the avm_kernel_config area.
#
peh@vidar:/tmp/YourFritz>

---

EDIT: I've found the reason for the "Unexpected end of input" from unpack_kernel.sh - I haven't really copied the last block of 4096 byte, because 1565952 is only adjusted on a 256 byte boundary instead of 4096 and it's necessary to add one more block, after dividing the offset (for the beginning of the embedded SquashFS image) by 4096. The correct command would be:

peh@vidar:/tmp/YourFritz> dd if=1750E_07.27.image of=kernel.image bs=4096 count=$(( ( 1565952 / 4096 ) + 1 ))
383+0 records in
383+0 records out
1568768 bytes (1.6 MB, 1.5 MiB) copied, 0.00581892 s, 270 MB/s
peh@vidar:/tmp/YourFritz> bash avm_kernel_config/unpack_kernel.sh < kernel.image > /dev/null
peh@vidar:/tmp/YourFritz>

... no more errors yet.

 

[baribal]

Hi @PeterPawn,

Many thx for such detailed instruction. Please find attached avm_kernel_config_area files from both firmware for DVB-C I have - 7.01 release and 7.08 beta. And they differs. I recall you said they should be equal for each device? What should I do next and which of both files should I use (I guess the one from the newer firmware 7.08)?

PS Just a remark I couldn't compile your programs using those sources http://osp.avm.de/fritzwlan/fritzwl...es-FRITZ.Box_WLAN_Repeater_DVB_C-07.01.tar.gz. However I managed to do it with the sources you used 7.27 1750E so I think this is normal due to you've made the patch for this source version explicitly.

 

[PeterPawn]

The FDT parts should be the same, indeed. All other parts are to be expected as different - if they would be always the same, there would be no needs for the avm_kernel_config extraction.

I have no idea, why the FDTs are different, too. The first assumption would be a different sort order of the entries - to know it for sure, it would be necessary to "decompile" both FDTs. But finally it doesn't matter, whether they're (functionally) different or not - you have to select one version to be included into the 1750E configuration area. If you realize later, that the selected version will not help, you may try the other one.

The attached file is a merge of the 1750E 07.27 area with the (only one) HWSubrevision 0 FDT for the DVB-C model (from 07.01) - this should be used to overwrite the generated file during the Freetz-NG build.

But you have to make sure, that all other variables from Freetz-NG are correctly set to run the avm_kernel_config part from Freetz-NG (with replacement of the generated file after the bin2asm call, mentioned some posts earlier in this thread) and to compile/link an own kernel - as I wrote above, this option (usually known as "Replace kernel" - or for short: RK) isn't activated yet for neither of both models in Freetz-NG.

Please keep in mind to run all builds for Freetz-NG from fmake and to save the created log files - even if you don't attach them immediately, it's highly probable they'll get requested later.

 

[baribal]

@PeterPawn

Thank you suggestion but unfortunately I can't find where to put this file (I can't find anything with such name after the image built for 7.27 1750E in freetz-ng):

freetz@freetz:~/freetz-ng$ find ./ -name "avm_kernel_config_area*"
freetz@freetz:~/freetz-ng$

I've set FREETZ_AVM_PROP_KERNEL_CONFIG_AREA_SIZE_0=n inside the ".config"

 

[PeterPawn]

I've set FREETZ_AVM_PROP_KERNEL_CONFIG_AREA_SIZE_0=n inside the ".config"

Why do you think, that's the right way? Can you find any other line containing =n in this file?

But there are other variables involved in the decision, whether "Replace kernel" will be working ... one more would be FREETZ_REPLACE_KERNEL (and FREETZ_REPLACE_KERNEL_AVAILABLE) or any selected additional loadable kernel module (LKM) with a FREETZ_MODULE_* setting, because without any of those settings the Linux kernel isn't selected as a make target: https://github.com/Freetz-NG/freetz...60fffac6eba11c65ad/make/linux/Makefile.in#L16 and therefore your changes to FREETZ_AVM_PROP_KERNEL_CONFIG_AREA_SIZE_0 setting would be ineffective either.

Conclusion: It's not as pretty easy to change Freetz-NG correctly to support RK for a special model - if you haven't a deep understanding, how Freetz (and Freetz-NG) works, you're facing a tough time and a big challenge. I had never said any others ... I had simply assumed, that this part was working already and only your attempt to enforce another FDT would be the problem.

I'm not interested in collecting all the changes needed to make this work in Freetz-NG - I don't use Freetz-NG either and I do not own any device of both models. For models like those, where the hardware layout is a different one (7520 vs. 7530 is completely different case, because here the layout is the same and AVM's kernel works on both models out of the box), the "alien" support from Freetz(-NG) isn't working, as long as no "Replace kernel" option is supported.

---

But if Freetz-NG doesn't support "Replace kernel" yet for these models, it's simpler with a second option I've mentioned in (meanwhile) #30:

But that's even the version, where you really need to replace the kernel with your own version - compiled from scratch and linked yourself. Another option (nothing for someone with lack of courage, but with a result that's much closer to the original kernel) is it, to unpack the 1750E kernel, patch the contained FDT area from AVM's kernel configuration (it doesn't contain any absolute addresses, iirc) and (re-)pack the kernel.

It's nothing for wimps ... but you know meanwhile (from here: https://www.ip-phone-forum.de/threads/freetz-für-avm-dvb-c-repeater-verfügbar.307070/post-2440340) where the configuration area starts in the unpacked kernel image for 1750E with FRITZ!OS 07.27 - so it's possible to simply patch the binary file (using a hexeditor) and to pack it again afterwards. I have only a backup of the 07.08 inhouse firmware for the DVB-C repeater - so I can only extract this version of FDT data myself.

The result (the patched and packed kernel image) may be used to replace the original kernel image - and the only remaining decision is, whether to modify the filesystem image, too. There are some changes (sometimes) required, because the HWRevision value from urlader environment will be taken to build some URLs. But these changes shouldn't be necessary to run the firmware (and to activate WLAN), as long as the WLAN settings (from /etc/default.Fritz_Box_HW206/avm/wlan*) don't differentiate between the affected HWRevision values for the "source" and the "sink" model.

---

Beside Freetz-NG there are two options to solve the HWRevision problem, if needed ... either to change the value from urlader environment to the 1750E value and to keep the filesystem image unchanged or to change the right places in some files to the (fixed) HWRevision value of the DVB-C model. If it''s possible to make persistent changes to the HWRevision value in the DVB-C urlader environment, this would be the easiest option - but it has to be tested by someone, who has access to such a device. Nevertheless any automatic update will not work - there are additional difference in the signature keys and some other values between the 1750E and DVB-C firmware (not to forget the needed changes to the kernel). And (useless to mention it explicitly) the DVB-C part will not work anymore, if a 1750E firmware will be used - there's no option to reactivate the DVB-C support, due to missing files in the 1750E firmware.

 

[PeterPawn]

There's currently a problem with GitHub comments for me - but it looks like, whether it's a more global error by GitHub - neither my Firefox 91 (Windows) nor any GitHub app from Android is just functioning.

Update: It was a bigger blackout: https://www.githubstatus.com/incidents/rmfrw9dfbtbp

But the last comment on GitHub (it's only an hour old):

but the tool still cant handle this kernel

(https://github.com/Freetz-NG/freetz-ng/issues/15#issuecomment-896084895) is obviously wrong after the update made yesterday.

As shown in this thread (https://www.ip-phone-forum.de/threads/freetz-für-avm-dvb-c-repeater-verfügbar.307070/post-2440340) it's possible to extract data and generate assembler sources with the updated version.

 

[baribal]

@PeterPawn

Thank you for details. I managed to patch the unpacked kernel in the hexeditor. How to pack the kernel again and create the new kernel.image? Packing the kernel is just "lzma -c"? I used fwmod tool to unpack the firmware. Shall I use it also for packing again?

 

[PeterPawn]

Are you sure, your changes are correct and complete? Did you check it with extract_avm_kernel_config and gen_avm_kernel_config utilities again?

Please attach the (freshly) extracted configuration area (as .zip file best) and the generated assembler source to your next post.

And no, packing a kernel isn't only a lzma call - there's an utility in the Freetz toolchain to be used: https://github.com/Freetz-NG/freetz-ng/tree/master/tools/make/lzma2eva-host

The parameters to call it (load address and entry point of the kernel) are shown by the unpack-kernel script: https://github.com/Freetz-NG/freetz-ng/blob/master/tools/unpack-kernel

You have to split the original image from AVM (not the outer TAR file, the inner kernel.image) into the kernel and the filesystem part - the location of sqsh signature is the start of filesystem image. Then you have to copy together your new kernel image (padded to a 256 byte boundary) and the filesystem image - the resulting file has to be installed using EVA. And I don't know, which partition is the right target - I'd think, it's MTD1. But that's only a guess (based on other devices) and you should make sure, whether it's correct.

EDIT: And by the way ... you should create a patch file for the hexadecimal changes using bsdiff: https://linux.die.net/man/1/bsdiff - this could be applied from a shell script (which automates all the needed steps) later.

 

[baribal]

@PeterPawn

Please see attached.

 

[PeterPawn]

There was a misunderstanding ... only the FDT part of DVB-C firmware has to be copied to the (binary) 1750E kernel configuration area. The attached assembler file contains the module list and version info from the 07.08 inhouse firmware for DVB-C repeaters - if this file was generated from the binary file provided, it's wrong for sure.

You have to take the 1750E kernel file, locate the configuration area at offset 0x468000 and change the list of entries (to remove HWSubrevisiion 1 and 7 - simply move the last entry with tag 0x107 up to the new end of array) and - last but not least - the memory address for the HWSubrevision 0 entry to the former value of HWSubrevision 7, because this is the first entry stored in the area (at offset 0x468040) and if it gets overwritten, the DVB-C FDT should fit into the available space in front of the version info. Then you may copy the FDT part from DVB-C binary area to the correct offset in the unpacked 1750E kernel - but I wouldn't use a hexeditor here anymore, because it's a heavy load of data and this may be copied using dd much better. A hexeditor is only needed to change the array list at offset 0x468010, because the new data for this area aren't available from other files.

 

[baribal]

@PeterPawn

It looks like FTD area of the dvb-c is larger than we have inside 1750E and it overwrites some important memory blocks in the 1750E. Please see attachment (in red you see the FTD from dvb-c copied to the unpacked kernel 1750E, and the last screenshot shows there is some data in this area which was overwritten ex. /lib/ld-uClibc.so.0 etc.).
So your program segment faults while trying to disassemble this bin file:

freetz@freetz:~/YourFritz/avm_kernel_config$ ./gen_avm_kernel_config 1750E-07.27_patched_bspatch_avm_kernel_config.bin > avm_kernel_config_area.1750E-07.27_patched_bspatch
Segmentation fault (core dumped)

 

[PeterPawn]

No, you've a wrong concept of avm_kernel_config area contents.

The only part, which has to be copied, is the "flattened device tree" for HWSubrevision 0 ... and the red area from your first screenshot shows the module memory section instead. This section is version-specific - the same applies (as the name says already) to the version section.

I've extracted the DVB-C configuration area from the inhouse version myself - it starts as follows:

00000000  80 4c 30 10 00 00 00 00  00 00 00 00 00 00 00 00  |.L0.............|
00000010  00 00 00 01 80 4c 4d 80  00 00 00 02 80 4c 4c c0  |.....LM......LL.|
00000020  00 00 00 05 80 4c 30 30  00 00 01 07 00 00 00 00  |.....L00........|
00000030  d0 0d fe ed 00 00 1c 8e 00 00 00 38 00 00 19 1c |...........8....|

The red part is the module memory table, it starts at 0x804c4d80 - obviously the highest memory address of all three entries ... one with tag 1, another one with tag 2 (it's the version info - marked with green at 0x804c4cc0) and the third is the FDT entry for HWSubrevision 0 (with tag 5, blue color and a memory address of 0x804c3030). The last entry (marked with yellow color) is the end of array marker with tag x0107 and no memory address (it's the end, not one more entry with data).

At offset 0x30 of the configuration area (it's located at memory address 0x804c3030 - the whole configuration area starts at memory address 0x804c3000 for this firmware version, this is the file offset 0) the first 8 byte of the FDT will be found. It's the FDT magic value (0xd00dfeed) and the overall length of the FDT (marked with gray) - so it's to expect, that the next entry (the version table at 0x804c4cc0) starts after offset 0x30 + 0x1c8e (= 0x1cbe) at the next 4-byte alignment (= 0x1cc0). And what starts there ... at memory address 0x804c4cc0 (it's 0x1cc0 + base address of 0x804c3000)? Correct, it's the version table, which has to be kept the one from the 1750E configuration area.

Now the FDT may be extracted from the configuration area with something like this:

peh@vidar:/tmp/YourFritz/avm_kernel_config> dd if=avm_kernel_config_dvb-c.bin of=dvb-c.fdt.bin bs=1 skip=$(( 0x30 )) count=$(( 0x1c90 ))
7312+0 records in
7312+0 records out
7312 bytes (7.3 kB, 7.1 KiB) copied, 0.0552657 s, 132 kB/s
peh@vidar:/tmp/YourFritz/avm_kernel_config>

This file has to be used to replace the FDT in the unpacked 1750E kernel at offset 0x468040 - it's length is the size of our FDT structure, padded to the next 4-byte boundary. To apply this over the unpacked kernel file (from 1750E 07.27), a dd command like this could be used:

peh@vidar:/tmp/YourFritz> dd if=avm_kernel_config/dvb-c.fdt.bin of=kernel.unpacked bs=1 seek=$(( 0x468040 )) conv=notrunc
7312+0 records in
7312+0 records out
7312 bytes (7.3 kB, 7.1 KiB) copied, 0.0531312 s, 138 kB/s

Now we have replaced the very first FDT entry in the 1750E kernel:

peh@vidar:/tmp/YourFritz> hexdump -C kernel.unpacked | grep -A 20 "^00468000"
00468000  80 4c 80 10 00 00 00 00  00 00 00 00 00 00 00 00  |.L..............|
00468010  00 00 00 01 80 4c de a8  00 00 00 02 80 4c dd e8  |.....L.......L..|
00468020  00 00 00 05 80 4c be b0  00 00 00 06 80 4c 9f 78  |.....L.......L.x|
00468030  00 00 00 0c 80 4c 80 40  00 00 01 07 00 00 00 00  |.....L.@........|
00468040  d0 0d fe ed 00 00 1c 8e  00 00 00 38 00 00 19 1c  |...........8....|
00468050  00 00 00 28 00 00 00 11  00 00 00 10 00 00 00 00  |...(............|
00468060  00 00 03 72 00 00 18 e4  00 00 00 00 00 00 00 00  |...r............|
00468070  00 00 00 00 00 00 00 00  00 00 00 01 00 00 00 00  |................|
00468080  00 00 00 03 00 00 00 0a  00 00 00 00 71 63 61 2c  |............qca,|
00468090  61 74 68 37 39 00 00 00  00 00 00 03 00 00 00 04  |ath79...........|
004680a0  00 00 00 0b 00 00 00 01  00 00 00 03 00 00 00 04  |................|
004680b0  00 00 00 1a 00 00 00 01  00 00 00 03 00 00 00 18  |................|
004680c0  00 00 00 26 41 56 4d 20  46 72 69 74 7a 52 65 70  |...&AVM FritzRep|
004680d0  65 61 74 65 72 20 44 56  42 2d 43 00 00 00 00 01  |eater DVB-C.....|
peh@vidar:/tmp/YourFritz>

and the last task is it, to change the array (marked in red) to remove the two unused HWSubrevision entries. The only needed entry for HWSubrevision 0 has a tag of 5, but needs the correct memory address - it's the former address of FDT for HWSubrevision 7. At the same time the end of array marker has to be moved up two entries (each of 8 byte).

Finally the unpacked kernel image has to look like this at offset 0x468000:

00468000  80 4c 80 10 00 00 00 00  00 00 00 00 00 00 00 00  |.L..............|
00468010  00 00 00 01 80 4c de a8  00 00 00 02 80 4c dd e8  |.....L.......L..|
00468020  00 00 00 05 80 4c 80 40  00 00 01 07 00 00 00 00  |.....L.......L.x|
00468030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |.....L.@........|
00468040  d0 0d fe ed 00 00 1c 8e  00 00 00 38 00 00 19 1c  |...........8....|

- now it has to be packed again.

 

[baribal]

@PeterPawn

Many thanks. I extracted the akc from the final patched kernel.

I built the tools from freetz-ng. Unpacked the kernel via unpack-kernel and get those settings:

TI_AR7.LoadAddress=0x80060000
TI_AR7.EntryAddress=0x80064870
TI_AR7.RecordLength=0x0017E3FE
TI_AR7.RecordChecksum=0x73FC5A97
WARNING: calculated TI-record checksum (0x7FEE0597) does not match the saved one (0x73FC5A97)
TI_AR7.LzmaCompressedStreamLength=0x0017E3E6
TI_AR7.LzmaUncompressedStreamLength=0x004C1F53
TI_AR7.LzmaCompressedStreamChecksum=0x1005902B

Now I need to lzma pack it, then use lzma2eva tools and combine it with the kernelsquashfs.raw:

tools/lzma e ~/kernel.raw.unpacked2 ~/kernel.raw.patched
tools/lzma2eva 0x80060000 0x80064870 ~/kernel.raw.patched ~/kernel.raw.patched.eva
cat ~/kernel.raw.patched ~/kernel.raw.patched.eva >kernel.image

Am I correct?

@PeterPawn

You said that the attached file is invalid? It is not the bsdiff but the pure bin file of the avm_kernel_config extracted by your tool from the final patched kernel. And an additional disassemble file from it.

"To verify your steps, other files would be more important - e.g. the extracted configuration area (in binary format) and the generated assembler code ... but both from the already patched kernel."

But that is exactly what this archive consists from.

@PeterPawn
"Looking at your planned commands ... this can't be the right order, because there have data to be appended in front of and after the packed data and your cat sequence would only append something. I'd bet, the lzma2eva tool creates the final file already."

tools/lzma2eva
usage: lzma2eva <loadadddr> <entry> <lzmafile> <evafile>

So it looks like lzma2eva requires lzma compressed file on input. Btw, the compressed kernel should be 256 byte padded (mean its size value must be able to divide to 256?)

I found here:

help/howtos/development/flash – Freetz

freetz.github.io

"All kernels that require EVA start with the hex sequence 81 12 ED FE , regardless of the endian. A second signature from offset 12 (0x0C) signals the type of compression of the following kernel data. The hex sequence 01 02 5A 07 means lzma "

 

[PeterPawn]

Am I correct?

Maybe ... but here's a point, were you can't do anything wrong. If you pack the kernel and immediately try to unpack it again and this gives you the same result as from the original one ... then you were obviously right.

I don't have in mind, how Freetz(-NG) handles packing of a kernel at all ... I would have to research it first, too, to give you a proper answer. I only know, that the (lzma-packed) kernel needs a correct prefix and suffix structure for it's unpacking by EVA and how both have to look like.

So it's easier, you try it yourself ... if it's the wrong command order, I may look for the right one even later (if you ask again). It it's already the right one ... congratulations. My claim is usually to provide hints for someone to get able to help himself.

@baribal:
EDIT: Looking at your planned commands ... this can't be the right order, because there have data to be appended in front of and after the packed data and your cat sequence would only append something at the end. I'd bet, the lzma2eva tool creates the final file already.

EDIT2:
The file attached to #59 is invalid - but the binary data from bsdiff is even not suited to be viewed by someone. It may be used to patch the original kernel with the new configuration area - nothing else.

To verify your steps, other files would be more important - e.g. the extracted configuration area (in binary format) and the generated assembler code ... but both from the already patched kernel.