Re: [Libunwind-devel] Remote unwinding support for perf on MIPS

[Luke Diamand]

On 30 July 2018 at 18:37, Sapir Natan <address@hidden> wrote:
> Hi Luke,
>
> Thank you so much for your detailed reply.
>
> We have the '.eh_frame' in our code.
> We also located a problem when reaching the function 
> 'perf_evsel__parse_sample' in file /tools/perf/util/evsel.c (linux 3.10)
>
> We noticed that the field 'data->user_stack.size' is assign the value saved 
> at 'array += size / sizeof(*array)' which is 0 We changed the line to use the 
> value stored in 'size' (which we checked to be the valid stack size), and now 
> 'perf' seems to create a valid callgraph This might be an issue in MIPS 
> architecture, while all other supported architectures contain a valid 
> argument at that offset (although I should note this is only a speculation at 
> this point)
>
> The fix on line 1176:
> data->user_stack.size = *array; // Old line
> data->user_stack.size = size;  // New line

I think I may have a somewhat newer version of the code - I think you
said you were using 3.10; I've got that kernel version but I also took
a bunch of patches to bring the perf support up-to-date.

The code I have says:

     data->user_stack.size = *array++;

That change came in with a change from Adrian Hunter in 54bd269 (Linux
kernel) so that may explain the difference?

But I'm using MIPSEL32, so perhaps that makes a difference? Or perhaps
this is the explanation for my other missing data!

Luke


>
> Have you encountered any problems with this code section?
> Do you think it might be a register issue on MIPS?
>
> Thanks,
> Sapir
>
> -----Original Message-----
> From: Luke Diamand [mailto:address@hidden
> Sent: Monday, July 30, 2018 11:57 AM
> To: Sapir Natan
> Cc: Sergey Korolev; Chen Muchtar; Roi Chen; address@hidden
> Subject: Re: [Libunwind-devel] Remote unwinding support for perf on MIPS
>
> On 30 July 2018 at 07:10, Sapir Natan <address@hidden> wrote:
>> Hi Luke,
>>
>>
>>
>> Thank you very much for your reply.
>> I am afraid I haven’t seen ‘_UPT_access_mem.c’ or ‘_UPT_reg_offset.c’
>> linked to perf or called from it at all (or anything from libunwind-ptrace).
>>
>> It is also already part of version 1.3-rc1, which means my perf still
>> fails to unwind even with this latest fix.
>>
>
> There are a ton of things that can go wrong along the way! I'm on vacation at 
> the moment and I don't have easy access to the relevant source code, so this 
> is all a bit from memory.
>
> I used the 3 patches I think you mentioned to add perf MIPS support.
>
> I've been getting to the point of capturing userspace callstacks (DWARF, not 
> fp) using perf+libunwind on MIPSEL32, and it appears to work reasonably well, 
> although I do see some callstacks possibly failing (most do, just a few 
> mysteriously missing).
>
> You need to make sure the code you are testing actually has any unwind tables 
> in it - check the .eh_frame section is present.
>
> I am using the current version of libunwind (1.3rc1 or thereabouts) and I 
> don't think it will work with older versions. You need to make sure that you 
> have the fix "dwarf: make dwarf_find_debug_frame public"
> as otherwise perf just won't use libunwind at all.
>
> I also found that in tdep-mips/libunwind_i.h, around line 150,
> read_s32() seems to do some weird stuff with the address, which I think is 
> for handling big-endian 64 bit addresses, but it doesn't seem to be the right 
> thing to do on MIPSEL32. For now, I've just hacked out the calls to 
> read_s32() and that seems to make it work - but you're actually on a 64 bit 
> target so it might be different for you. I need to create a proper patch 
> (sorry, not had time recently).
>
> I also had no end of fun with split-debug (as generated by
> OpenEmbedded) because the debug files end up in a place that libunwind does 
> not expect to find them in. Watch out for that.
>
> Another thing you will find is that with your version of perf, if you are 
> running the unwinding on an x86-64 host, then most/all versions of perf will 
> try to unwind the callstacks as though they are x86-64 callstacks rather than 
> MIPS, which obviously doesn't work. From about linux 4.7 onwards there is 
> cross-platform perf unwinding available, but the hooks for MIPS support have 
> not been added (I don't yet have working hooks either). There are a couple of 
> ways around this: you can run "perf report" on the target (but that is very 
> slow) or you can run "perf report" on your x86-64 host under qemu-mips (you 
> want the userland emulator not the full system version). This is what I've 
> mostly been doing.
>
> Make sure that libunwind has been built with --enable-debug-frame otherwise 
> it won't work if your callstack goes through any code that was not compiled 
> with -fasynchronous-unwind-tables or -fexceptions.
>
> If you run "perf report -D" then it dumps out the file contents. Look through 
> to see if you have any register dumps and check they are convincing looking 
> MIPS register values.
>
> Also look for the stack captures in the "perf report -D" report.
> That's the giant 8kByte samples. If you haven't got any, then it's not doing 
> the DWARF sampling at all.
>
> Also check for the MMAP sections in the perf capture - sometimes perf can't 
> read the /proc/PID/maps file of the thing it is trying to capture.
>
> It sounds to me a bit as though you don't have a .eh_frame section in your 
> image (there's a linker option as well that you need), and if perf/libunwind 
> doesn't have access to the debug image, it won't work.
> But it could also be quite a few other things!
>
> Good luck!
>
> Luke
>
>
>>
>>
>> Did this patch fix the perf unwinding on MIPS issue for you?
>>
>> If so, can you tell me which patch you used to add this capability to
>> perf on MIPS? (patch to the linux code)
>>
>> Is there anything I’m missing here?
>>
>>
>>
>> Thanks,
>>
>> Sapir
>>
>>
>>
>> From: Luke Diamand [mailto:address@hidden
>> Sent: Sunday, July 29, 2018 8:59 PM
>> To: Sergey Korolev
>> Cc: Sapir Natan; Chen Muchtar; Roi Chen; address@hidden
>> Subject: Re: [Libunwind-devel] Remote unwinding support for perf on
>> MIPS
>>
>>
>>
>> I had a similar problem on mipsel32. At least on that the problem was
>> that the accessor function was calling read_s32 which seems to do the
>> wrong thing on that platform. Fixing that seems to have made it work.
>> I'm in the process of trying to organise a patch.
>>
>>
>>
>> On Sun, 29 Jul 2018, 18:25 Sergey Korolev, <address@hidden> wrote:
>>
>> In my opinion a mentioned patch [1] does not work properly
>>
>> since it defines _UPT_reg_offset elements as
>>
>>
>> +    [UNW_MIPS_R0]  =  0,
>> +    [UNW_MIPS_R1]  =  1,
>> +    [UNW_MIPS_R2]  =  2,
>> +    [UNW_MIPS_R3]  =  3,
>> +    [UNW_MIPS_R4]  =  4,
>> +    [UNW_MIPS_R5]  =  5,
>> +    [UNW_MIPS_R6]  =  6,
>> +    [UNW_MIPS_R7]  =  7,
>> +    [UNW_MIPS_R8]  =  8,
>> +    [UNW_MIPS_R9]  =  9,
>> +    [UNW_MIPS_R10] = 10,
>> +    [UNW_MIPS_R11] = 11,
>> +    [UNW_MIPS_R12] = 12,
>> +    [UNW_MIPS_R13] = 13,
>> +    [UNW_MIPS_R14] = 14,
>> +    [UNW_MIPS_R15] = 15,
>> +    [UNW_MIPS_R16] = 16,
>> +    [UNW_MIPS_R17] = 17,
>> +    [UNW_MIPS_R18] = 18,
>> +    [UNW_MIPS_R19] = 19,
>> +    [UNW_MIPS_R20] = 20,
>> +    [UNW_MIPS_R21] = 21,
>> +    [UNW_MIPS_R22] = 22,
>> +    [UNW_MIPS_R23] = 23,
>> +    [UNW_MIPS_R24] = 24,
>> +    [UNW_MIPS_R25] = 25,
>> +    [UNW_MIPS_R26] = 26,
>> +    [UNW_MIPS_R27] = 27,
>> +    [UNW_MIPS_R28] = 28,
>> +    [UNW_MIPS_R29] = 29,
>> +    [UNW_MIPS_R30] = 30,
>> +    [UNW_MIPS_R31] = 31,
>> +    [UNW_MIPS_PC]  = 64,
>>
>>
>>
>> but a code in _UPT_access_mem.c expects that _UPT_reg_offset
>>
>> is a byte offset, and not a word offset or a register index.
>>
>>
>>
>> [1]
>> http://git.savannah.gnu.org/gitweb/?p=libunwind.git;a=commit;h=2a5d1a6
>> 29634b3225fccb1ed8a0a765bccc77f2e
>>
>>
>>
>> On Sun, Jul 29, 2018 at 6:18 PM, Sapir Natan <address@hidden> wrote:
>>
>> Hi,
>>
>>
>>
>> We are trying to obtain a callgraph using ‘perf’ utility, while
>> running on MIPS architecture with linux 3.10 (64Bit processes)
>>
>> We wish to obtain callgraph on:
>>
>> ·         Terminated processes
>>
>> ·         Running processes (using –p <pid>)
>>
>>
>>
>> Following the resolution that perf is unable to create callgraph on
>> MIPS using frame pointer, we tried using libunwind
>>
>> Since ‘dwarf’ option for perf isn’t part of linux 3.10 (or any other
>> version), we added a patch (composed of the following 3 patches):
>>
>> ·
>> https://www.linux-mips.org/archives/linux-mips/2013-05/msg00123.html
>>
>> ·
>> https://www.linux-mips.org/archives/linux-mips/2013-05/msg00115.html
>>
>> ·
>> https://www.linux-mips.org/archives/linux-mips/2016-04/msg00003.html
>>
>> ·         As part of the patch we also linked ‘perf’ with ‘libunwind’ (open
>> source version 1.2.1).
>>
>>
>>
>> This patch added the ‘dwarf’ capability to ‘perf’ on MIPS, however we
>> still couldn’t obtain a usable callgraph (most functions had no trace)
>>
>> The problem we encountered was that when reaching ‘maps__find´
>> function (linux/tools/perf/util/map.c), the binary search for the
>> required entry failed (could not be located)
>>
>> From our investigation, we can support local unwinding but not remote
>> (which is the type of unwinding required by ‘perf’)
>>
>>
>>
>> Checking online we saw on ‘libunwind’ NEWS: ‘** Mips remote unwind support’
>> for version 1.3.
>>
>> We replaced our ‘1.2.1’ version with ‘1.3-rc1’ and re-checked, however
>> the results remained the same (no callgraph).
>>
>>
>>
>> Could you please assist?
>>
>> ·         What was the issue you encountered that the fix in 1.3 version
>> resolved?
>>
>> ·         Can the new addition enable ‘perf’ to support callgraph on MIPS?
>>
>> ·         Did we miss anything in our steps (as described above)?
>>
>> ·         Can you advice on the next steps?
>>
>>
>>
>> Thanks,
>>
>> Sapir
>>
>>
>>
>>
>> _______________________________________________
>> Libunwind-devel mailing list
>> address@hidden
>> https://lists.nongnu.org/mailman/listinfo/libunwind-devel
>>
>>
>>
>> _______________________________________________
>> Libunwind-devel mailing list
>> address@hidden
>> https://lists.nongnu.org/mailman/listinfo/libunwind-devel
>>
>>
>>
>> Email secured by Check Point.
>
> Email secured by Check Point.