Re: [Gnewsense-dev] Debugging linux freeze / crash problems on the yeeloong

[Wu Zhangjin]

Hi,

On Tue, 2010-01-05 at 16:00 -0500, Danny Clark wrote:
> Could someone speak to how to debug problems on the yeeloong, where the
> kernel, linux freezes without spewing any errors to the console?

are you working with the console? and with the 2D acceleration of sm712
enabled? If yes, the problem should be the bug of 2D acceleration.

We have found a bug in the 2D acceleration of sm7xx video driver, it may
make the whole system hang when scrolling the screen, for example, if we
copy some files from another machines to yeeloong via scp and the screen
output is enabled, the whole system may hang.

Before this bug is really fixed, please ensure CONFIG_FB_SM7XX_ACCEL is
disabled:

$ gunzip < /proc/config.gz | grep SM7XX
CONFIG_FB_SM7XX=y
# CONFIG_FB_SM7XX_ACCEL is not set

> 
> I found Wu's paper at
> http://lwn.net/images/conf/rtlws11/papers/proc/p14.pdf and we are trying
> ftrace,

If you want to play with ftrace, there are two good documents from the
original author of ftrace:

Debugging the kernel using Ftrace - part 1
http://lwn.net/Articles/365835/
Debugging the kernel using Ftrace - part 2 (subscriber-only)
http://lwn.net/Articles/366796/

>  but it would be useful to know exactly what tools Lemote used
> early in the bring-up process - for example people have mentioned that
> it is possible to do linux debugging using PMON, but I've been unable to
> find any documentation on how to do that.
> 

You can get the documentation from

PMON> h

find out the Debugger and Memory part, which will give you some help.

> One specific thing we would like to know is if anyone has gotten kgdb to
> work on the system - Bernie thought that maybe at the momment mips big
> endien was supported, but mips little endian (mipsel), as the loongson2f
> is, was not.

kgdb really works, besides, serial port debugging is a good choice when
the system is in the progress of booting(but for YeeLoong, it should not
be a good choice, except we "break" the machine and touch the serial
port pins ourselves):

# early printk will help a lot before the console is initialized
CONFIG_EARLY_PRINTK=y

# serial port driver is useful with console=ttyS0,115200 ... 
CONFIG_SERIAL_8250=y

and also, netconsole is another choice(after the network is
initialized):

Device Drivers -->
            
[*] Network device support  -->
[*] Network console logging support

and the watch register(the 18th register of cp0) of loongson can be used
to trace the read/write to a virtual address:

                            63  3   2  1  0
                            VADDR,  0, R, W

Set(whath an address）: load t0, (VADDR | 0RW); mtc0 t0, 18
Clear: mtc0 $ZERO, 18

and there are also some other methods, including the kernel APIs: BUG,
WARN_ON, show_registers, show_regs, dump_stack/show_trace/show_stack,
print_symbol, these APIs may help to debug the kernel oops.

and the "magic sysrq" may help to debug the system freeze, the oprofile,
ftrace and kgcov may help to improve the performance of the system.

Regards,
        Wu Zhangjin