Can't build my custom freetz

[Silent-Tears]

I use this with my native Ubuntu-Linux on a core2duo, and an AMD phenonX4, with a native openSuSE, too.

 

[SystemR89]

I don't know if it's important but I use a 32bit version of Debian because there are an incompatibility with all 64 bit systems with my notebook (samsung P210/Q210)

 

[Silent-Tears]

Ok, this night I installed 2 versions of ubuntu server 8.10. One with 32bit, one other as a 64bit-system. Both are installed under vmware server 2, running on Windows Vista.
There is no problem with both versions. They have nearly the same config (32/64bitspecific), are both up-to-date, and i built firmwares for 7170 und 7270 without any trouble.
I followed the wiki-instructions, which packages are needed for building freetz-images.

 

[make]

Are you sure that you are using only 1 thread? The last lines of strace2.ttx suggest otherwise:

futex(0xade1f0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
+++ killed by SIGABRT +++

It seems that the program receives SIGABRT while it is in the futex() syscall.

I used the -p 1 option of unsquashfs3-lzma. I do not know how the program actually interprets this option.

Can you repeat it with "strace -f"? And I think ltrace also has the option -f.

I've attached the output of strace and ltrace both invoked with -f.

I also tried to build a 7170 firmware, but unsquashfs fails again.

To test the multicore idea I forced fedora to run single-core by setting maxcpus=1 as kernel argument. No change, unsquashfs is still aborting.

Going back to an older kernel (2.6.27.9), just to be sure, didnt help either.

At least using the unsquashfs3-lzma from freetz-stable instead the version referenced from trunk is still working fine.

Could it be related to my local build environment, say my gcc version? What gcc versions are you guys using? Anybody interested in trying to use my unsquashfs3-lzma in a vmware known to be able to build freetz? Anything left I could try?

 

[Silent-Tears]

Could it be related to my local build environment, say my gcc version? What gcc versions are you guys using? Anybody interested in trying to use my unsquashfs3-lzma in a vmware known to be able to build freetz? Anything left I could try?

I think the build-environment is the first place to look at, cause 32/64bit, single/multicore, trunk/stable, etc is not the point.

Could you please upload your unsquashfs-lzma somewhere ,that someone - maybe me - can test this?

 

[olistudent]

32/64bit, single/multicore, trunk/stable, etc is not the point.

I switched back to squashfs-3.3 in stable-1.1 because of this issue. In squashfs-3.4 a new, multithreaded version of unsquashfs was implemented.

Regards
Oliver

 

[RalfFriedl]

I've attached the output of strace and ltrace both invoked with -f.

The strace shows that multiple threads are created:

[pid 28672] [B]tgkill(28670, 28672, SIGABRT[/B] <unfinished ...>
[pid 28673] <... futex resumed> )       = 0
[pid 28672] <... tgkill resumed> )      = 0
[pid 28673] futex(0x753ee4, FUTEX_WAIT_PRIVATE, 3, {0, 249911359} <unfinished ...>
[pid 28672] --- SIGABRT (Aborted) @ 0 (0) ---
Process 28672 detached
[pid 28670] <... futex resumed> )       = 1
[pid 28673] +++ killed by SIGABRT +++
[pid 28671] +++ killed by SIGABRT +++
[pid 28674] +++ killed by SIGABRT +++
+++ killed by SIGABRT +++

The interesting thing ist that thread 28672 sends a SIGABRT only to itself, yet also the other threads show "killed by SIGABRT" but not the "SIGABRT (Aborted)" message from thread 28672. It seems that the threads intentionally sends itself this signal, most likely in reaction to some error.

The ltrace doesn't seem to recognize the option -f, it shows nothing of multiple threads. The option -S to display the syscalls would also be interesting, but less so if it doesn't show the threads.

Can you try gdb:

$ gdb /home/make/projects/freetz-trunk/tools/unsquashfs3-lzma
# gdb starts
(gdb) run -n -f -p 1 -dest build/original/filesystem build/original/kernel/kernelsquashfs.raw
# program should run and stop with SIGABRT
(gdb) where
# should show the call list

 

[olistudent]

Perhaps some special CFLAGS (-g) are needed to show this information?

 

[RalfFriedl]

Yes, adding -g and removing -O from CFLAGS makes the output more informative.

 

[make]

Ralf,

I recompiled unsquash3-lzma with -g but without -O. I also defined SQUASHFS_TRACE to get a more verbose output.

The ltrace doesn't seem to recognize the option -f, it shows nothing of multiple threads. The option -S to display the syscalls would also be interesting, but less so if it doesn't show the threads.

Find new ltrace and strace outputs attached.

Can you try gdb:

I ran unsquashfs3-lzma in the debugger as you have had suggested. The output is also attached. It seems that there is a code path where free(...) is called with an invalid pointer. gdb output is attached in gdb.zip

If you like you can also try to reproduce the problem by executing the binaries from unsquash3-lzma.zip. unsquash3-lzma is compiled with -g, unsquash3-lzma-g has both debug info and SQUASHFS_TRACE defined.

My gcc version (gcc -dumpversion) is 4.3.2.

 

[RalfFriedl]

I will have a look at it when I have a little more time.

Could you also (if you haven't already) create a statically linked unsquash3 program with -g and without -O, just in case it is a problem with your libraries. Of course you should verify that an error occurs also with this version.

 

[make]

Could you also (if you haven't already) create a statically linked unsquash3 program with -g and without -O, just in case it is a problem with your libraries. Of course you should verify that an error occurs also with this version.

Statically linked version with -g and SQUASHFS_TRACE, without -O is attached. Behaviour has not changed, uncompressing still fails.

The dynamically linked image is using the following libraries:

[make@mini freetz-trunk]$ ldd tools/unsquashfs3-lzma
        linux-vdso.so.1 =>  (0x00007fffecffe000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003625000000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x0000003626c00000)
        libm.so.6 => /lib64/libm.so.6 (0x0000003624c00000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003626000000)
        libc.so.6 => /lib64/libc.so.6 (0x0000003624400000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003624000000)
[make@mini freetz-trunk]$ ls -l /lib64/libpthread.so.0 /usr/lib64/libstdc++.so.6 /lib64/libm.so.6 /lib64/libgcc_s.so.1 /lib64/libc.so.6 /lib64/ld-linux-x86-64.so.2
lrwxrwxrwx 1 root root  9 2008-12-26 19:50 /lib64/ld-linux-x86-64.so.2 -> ld-2.9.so
lrwxrwxrwx 1 root root 11 2008-12-26 19:50 /lib64/libc.so.6 -> libc-2.9.so
lrwxrwxrwx 1 root root 28 2008-11-19 22:29 /lib64/libgcc_s.so.1 -> libgcc_s-4.3.2-20081105.so.1
lrwxrwxrwx 1 root root 11 2008-12-26 19:50 /lib64/libm.so.6 -> libm-2.9.so
lrwxrwxrwx 1 root root 17 2008-12-26 19:50 /lib64/libpthread.so.0 -> libpthread-2.9.so
lrwxrwxrwx 1 root root 19 2008-11-19 22:29 /usr/lib64/libstdc++.so.6 -> libstdc++.so.6.0.10

 

[olistudent]

Hi.
I can unsquashfs a filesystem with your attached versions.

static version

...
unsquashfs: create_inode: pathname build/original/filesystem//var.tar
unsquashfs: create_inode: regular file, file_size 20480, blocks 0
unsquashfs: write_file: regular file, blocks 0
unsquashfs: read_fragment: reading fragment 142
unsquashfs: read_bytes: reading from position 0x7d427e, bytes 763
unsquashfs: writer: regular file, blocks 1
unsquashfs: writer: regular file, blocks 1
created 1769 files
created 122 directories
created 257 symlinks
created 0 devices
created 0 fifos

dynamic version:

[root@fedora trunk]# ./unsquashfs3-lzma-g -n -d build/original/filesystem build/original/kernel/kernelsquashfs.raw
2190 inodes (2430 blocks) to write
 
created 1769 files
created 122 directories
created 257 symlinks
created 164 devices
created 0 fifos
[root@fedora trunk]#

The difference in devices is okay because I did unpack the image as root to avoid this error messages.

System:
Core2Duo T8300
Host: Windows XP SP3
VMWare Server 2.0-122956
Guest: Fedora 10

uname -a
Linux fedora 2.6.27.15-170.2.24.fc10.x86_64 #1 SMP Wed Feb 11 23:14:31 EST 2009 x86_64 x86_64 x86_64 GNU/Linux

 

[make]

Could you please try the statically linked version that I've just added? If that version fails on your system, the problem is somehow related to the libraries used to build the unsquashfs3-lzma binary.

 

[olistudent]

And if it doesn't fail?

Updated above post.

 

[make]

I guess this means that we still have not identified the real cause of the problem.

I recompiled unsquashfs with rmdebug (http://www.hexco.de/rmdebug/). Binary and output are attached.

I will be able to try more suggestions on friday.

 

[RalfFriedl]

Your static binary seems to work on my system, which in this case means there is no obvious error.
But running your rmdebug version, I get output similar to your, about memory corruption. The output is not really deterministic, most likely because the program always uses threads, independent of the number of decompression threads used. It is probably a thread synchronization issue or some other problem with the memory allocation.

Now the question is whether we should search for the cause of this or simply use version 3.3 and wait for a new version where this will hopefully be fixed.

 

[olistudent]

The unsquashfs in version 3.3 is not multithreaded. But I think with our small filesystems it will be no problem to reverse this change.
I can open a bug for it on squashfs bug tracker but first thing they will ask if it happens with an unpatched version too.

It's unclear to me why a thread synchronization issue appears only on some machines and this regularly.

@make
Perhaps you can build an unpatched version and create, unpack a filesystem with it?

Regards
Oliver

 

[RalfFriedl]

Can you build a version with electric fence?
From the description it should trap in the debugger directly at the offending instruction.

PS:
We don't really need multithreaded for our size of the firmware, although it would be nice.

 

[make]

I will have a look at electric fence on friday or saturday. rmdebug simply was the first library I found and it was easy to add.

I already tried the CVS head of unsquashfs ("4.0") but the changes are minor compared with the current version and the problem persists.