Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

In this Discussion

Success: Bring-up of LLVM/clang-built Linux ARM(32-bit) kernel for Android - Nexus 5

Success: Bring-up of LLVM/clang-built Linux ARM(32-bit) kernel for Android - Nexus 5

Hello,

I would like to share my successful bring-up of LLVM/clang-built Linux ARM(32-bit) hammerhead kernel for Android(Resurrection Remix/Oreo, including the latest "RR-O-v6.1.0-20180611-hammerhead-Official") running on my Nexus 5 smartphone. After having successfully brought up LLVM/clang-built Linux kernel(since v4.15.7 to the most recent v4.17) on x86_64, I was interested in accomplishing the same on the ARM platform of my Nexus 5 - Android smartphone. So, here is the complete report of the same for the interested people. 

The main advantage of the clang-built Android ARM(32-bit) hammerhead kernel for my Nexus 5 has been the better battery usage when compared to that of gcc-built kernel, with the same kernel config and hardware(my Nexus 5 Android Smartphone). Details of the same can be found below.

NOTE : By the way, I came across some reports of ARM64 clang-kernel for some Android Smartphones - but, the information over there did *not* help for my ARM32 clang-kernel case of Nexus 5(hammerhead). So, I started off this project from *scratch* and it has been lot of *entirely my own original work* to first successfully build the ARM32 clang-kernel for Nexus 5(hammerhead) and second to make it *actually work* on the real hardware - Nexus 5.

Cheers.


Android ARM(32-bit) clang-kernel bring-up for Nexus 5(hammerhead)

[Android Version Information] & [Battery Usage of a clang-built kernel ~ better than that of gcc-built kernel (shows one of the instances)]



[1] Android NDK r13b [LLVM/clang + binutils(as, ld, etc)] [2] Android NDK r17 [LLVM/clang + binutils(as, ld, etc)]



[3] Main LLVM/clang + Android NDK r13b binutils(as, ld, etc) [4] Main LLVM/clang + Android NDK r17 binutils(as, ld, etc)



[5] Snapdragon Qualcomm LLVM/clang + NDK r13b binutils(as, etc) [6] Snapdragon Qualcomm LLVM/clang + NDK r13b binutils(as, etc)

[Average Battery Usage]




[Contd.... below in next post - due to space restrictions per post]

Comments

  • BUILD SYSTEM INFORMATION


    Code:
    #### Build system information ####
    
    exp@exp:~$ 
    exp@exp:~$ sudo dmidecode -t system | grep "Manufacturer:\|Version:"
        Manufacturer: LENOVO
        Version: Lenovo Y50-70 Touch
    exp@exp:~$ 
    exp@exp:~$
    
    exp@exp:~$ 
    exp@exp:~$ sudo dmidecode -t processor | grep "Version\|Family:"
        Family: Core i7
        Version: Intel(R) Core(TM) i7-4710HQ CPU @ 2.50GHz
    exp@exp:~$ 
    exp@exp:~$
    
    exp@exp:~$ 
    exp@exp:~$ cat /proc/meminfo | grep MemTotal
    MemTotal:       16332968 kB
    exp@exp:~$ 
    exp@exp:~$ 
    
    exp@exp:~$ 
    exp@exp:~$  lsb_release -a
    No LSB modules are available.
    Distributor ID:    Ubuntu
    Description:    Ubuntu 17.10
    Release:    17.10
    Codename:    artful
    exp@exp:~$ 
    exp@exp:~$

    BUILD SUMMARY

    Code:
    #### Total build time ####
    
    
    38m6.816s
    
    #### Build times ####
    
    
    [GCC NDK r13b] : 4m15.596s
    [GCC NDK r17] : 4m13.983s
    [Android NDK r13b : LLVM/clang + binutils(ld and as)] : 4m3.665s
    [Android NDK r17 : LLVM/clang + binutils(ld and as)] : 4m4.683s
    [Main LLVM/clang + Android NDK r13b binutils(ld and as)] : 6m8.064s
    [Main LLVM/clang + Android NDK r17 binutils(ld and as)] : 6m3.457s
    [Qualcomm Snapdragon LLVM/clang + Android NDK r13b binutils(ld and as)] : 4m32.581s
    [Qualcomm Snapdragon LLVM/clang + Android NDK r17 binutils(ld and as)] : 4m44.779s
    LONGEST AND SHORTEST BUILD

    Code:
    ##### Longest build ####
    Name : Main LLVM/clang + Android NDK r13b binutils(ld and as)
    Time : 6m8.064s
    boot.img : boot-main-llvm-clang-ndk-r13b-binutils-ld-as.img
    zImage-dtb : zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as
    
    
    ##### Shortest build ####
    Name : Android NDK r13b : LLVM/clang + binutils(ld and as)
    Time : 4m3.665s
    boot.img : boot-ndk-r13b-clang-llvm-binutils-ld-as.img
    zImage-dtb : zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as

    LARGEST AND SMALLEST IMAGES

    Code:
         ◈ Largest boot img : boot-ndk-r17-gcc.img ❏ Size : 13M (12984320 bytes)
         ◈ Smallest boot img 1 : boot-main-llvm-clang-ndk-r17-binutils-ld-as.img ❏ Size : 11M (11272192 bytes)
         ◈ Smallest boot img 2 : boot-main-llvm-clang-ndk-r13b-binutils-ld-as.img ❏ Size : 11M (11272192 bytes)
    
    
         ◈ Largest zImage-dtb : zImage-dtb-ndk-r17-gcc ❏ Size : 12M (11844904 bytes)
         ◈ Smallest zImage-dtb : zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as ❏ Size : 9.7M (10132600 bytes)
    BOOT IMAGES SUMMARY

    Code:
         ◈ boot-ndk-r17-gcc.img ❏ Size ~ 13M (12984320 bytes)
         ◈ boot-ndk-r13b-gcc.img ❏ Size ~ 13M (12978176 bytes)
         ◈ boot-ndk-r13b-clang-llvm-binutils-ld-as.img ❏ Size ~ 13M (12625920 bytes)
         ◈ boot-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as.img ❏ Size ~ 12M (11610112 bytes)
         ◈ boot-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as.img ❏ Size ~ 12M (11610112 bytes)
         ◈ boot-ndk-r17-clang-llvm-binutils-ld-as.img ❏ Size ~ 11M (11476992 bytes)
         ◈ boot-main-llvm-clang-ndk-r13b-binutils-ld-as.img ❏ Size ~ 11M (11272192 bytes)
         ◈ boot-main-llvm-clang-ndk-r17-binutils-ld-as.img ❏ Size ~ 11M (11272192 bytes)

    ZIMAGE-DTB SUMMARY

    Code:
         ◈ zImage-dtb-ndk-r17-gcc ❏ Size ~ 12M (11844904 bytes)
         ◈ zImage-dtb-ndk-r13b-gcc ❏ Size ~ 12M (11837640 bytes)
         ◈ zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as ❏ Size ~ 11M (11487176 bytes)
         ◈ zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as ❏ Size ~ 10M (10469728 bytes)
         ◈ zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as ❏ Size ~ 10M (10469680 bytes)
         ◈ zImage-dtb-ndk-r17-clang-llvm-binutils-ld-as ❏ Size ~ 10M (10336624 bytes)
         ◈ zImage-dtb-main-llvm-clang-ndk-r17-binutils-ld-as ❏ Size ~ 10M (10132608 bytes)
         ◈ zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as ❏ Size ~ 10M (10132600 bytes)

    RAMDISK INFORMATION

    Code:
    Ramdisk(pre-built - from RR) : 
    
    
         ◈ boot.img-ramdisk.gz ❏ Size ~ 1M (1136400 bytes)


    [Contd.... below in next post - due to space restrictions per post]


  • Clang-KERNEL INFORMATION(from each of the zImage-dtb images)

    Code:
    exp@exp:~$ 
    exp@exp:~$ ../show_kernel_compiler_all.sh 
    #### Kernel compiler information ####
    
    NOTE : Analyzing the images by decompressing them based on lz4 magic (\x02\x21\x4c\x18) . . .
    
    Image metadata(zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as) : 
    +++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as
    +++ tail -1
    +++ cut -d: -f 1
    ++ pos1=5381
    ++ dd if=zImage-dtb-main-llvm-clang-ndk-r13b-binutils-ld-as bs=5381 skip=1
    ++ lz4 -d
    ++ eclang
    ++ head -1
    +++ strings -a
    +++ grep 'clang version'
    Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Flash clang version 7.0.332826 (https://git.llvm.org/git/clang 4029c7ddda99ecbfa144f0afec44a192c442b6e5) (https://git.llvm.org/git/llvm 1181c40e0e24e0cca32e2609686db1f14151fc1a) (based on LLVM 7.0.332826) - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:33:21 PDT 2018
    ++ set +x
    
    Image metadata(zImage-dtb-main-llvm-clang-ndk-r17-binutils-ld-as) : 
    +++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-main-llvm-clang-ndk-r17-binutils-ld-as
    +++ cut -d: -f 1
    +++ tail -1
    ++ pos1=5373
    ++ dd if=zImage-dtb-main-llvm-clang-ndk-r17-binutils-ld-as bs=5373 skip=1
    ++ lz4 -d
    ++ eclang
    ++ head -1
    +++ strings -a
    +++ grep 'clang version'
    Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Flash clang version 7.0.332826 (https://git.llvm.org/git/clang 4029c7ddda99ecbfa144f0afec44a192c442b6e5) (https://git.llvm.org/git/llvm 1181c40e0e24e0cca32e2609686db1f14151fc1a) (based on LLVM 7.0.332826) - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:39:25 PDT 2018
    ++ set +x
    
    Image metadata(zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as) : 
    +++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as
    +++ tail -1
    +++ cut -d: -f 1
    ++ pos1=5509
    ++ dd if=zImage-dtb-ndk-r13b-clang-llvm-binutils-ld-as bs=5509 skip=1
    ++ lz4 -d
    ++ eclang
    ++ head -1
    +++ grep 'clang version'
    +++ strings -a
    Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Android clang version 3.8.256229 (based on LLVM 3.8.256229) - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:23:08 PDT 2018
    ++ set +x
    
    Image metadata(zImage-dtb-ndk-r17-clang-llvm-binutils-ld-as) : 
    +++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-ndk-r17-clang-llvm-binutils-ld-as
    +++ tail -1
    +++ cut -d: -f 1
    ++ pos1=5449
    ++ dd if=zImage-dtb-ndk-r17-clang-llvm-binutils-ld-as bs=5449 skip=1
    ++ lz4 -d
    ++ eclang
    ++ head -1
    +++ strings -a
    +++ grep 'clang version'
    Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Android (4691093 based on r316199) clang version 6.0.2 (https://android.googlesource.com/toolchain/clang 183abd29fc496f55536e7d904e0abae47888fc7f) (https://android.googlesource.com/toolchain/llvm 34361f192e41ed6e4e8f9aca80a4ea7e9856f327) (based on LLVM 6.0.2svn) - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:27:14 PDT 2018
    ++ set +x
    
    Image metadata(zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as) : 
    +++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as
    +++ tail -1
    +++ cut -d: -f 1
    ++ pos1=5417
    ++ dd if=zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r13b-binutils-ld-as bs=5417 skip=1
    ++ lz4 -d
    ++ eclang
    ++ head -1
    +++ strings -a
    +++ grep 'clang version'
    Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Snapdragon LLVM ARM Compiler 4.0.2 for Android NDK (based on llvm.org 4.0+) - clang version 4.0.2 for Android NDK - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:43:58 PDT 2018
    ++ set +x
    
    Image metadata(zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as) : 
    +++ grep -P -a -b --only-matching '\x02\x21\x4c\x18' zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as
    +++ tail -1
    +++ cut -d: -f 1
    ++ pos1=5409
    ++ dd if=zImage-dtb-qualcomm-snapdragon-llvm-clang-ndk-r17-binutils-ld-as bs=5409 skip=1
    ++ lz4 -d
    ++ eclang
    ++ head -1
    +++ strings -a
    +++ grep 'clang version'
    Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Snapdragon LLVM ARM Compiler 4.0.2 for Android NDK (based on llvm.org 4.0+) - clang version 4.0.2 for Android NDK - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:48:42 PDT 2018
    ++ set +x
    
    exp@exp:~$ 
    exp@exp:~$


    Clang-KERNEL INFORMATION(from dmesg extracted from each of the boot instances)

    Code:
    exp@exp:~$ 
    exp@exp:~$ cat android1/android1_dmesg.txt | grep "clang\|Machine"
    [    0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Android clang version 3.8.256229 (based on LLVM 3.8.256229) - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:23:08 PDT 2018
    [    0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD
    exp@exp:~$ 
    exp@exp:~$ 
    exp@exp:~$ cat android2/android2_dmesg.txt | grep "clang\|Machine"
    [    0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Android (4691093 based on r316199) clang version 6.0.2 (https://android.googlesource.com/toolchain/clang 183abd29fc496f55536e7d904e0abae47888fc7f) (https://android.googlesource.com/toolchain/llvm 34361f192e41ed6e4e8f9aca80a4ea7e9856f327) (based on LLVM 6.0.2svn) - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:27:14 PDT 2018
    [    0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD
    exp@exp:~$ 
    exp@exp:~$ 
    exp@exp:~$ cat main1/main1_dmesg.txt | grep "clang\|Machine"
    [    0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Flash clang version 7.0.332826 (https://git.llvm.org/git/clang 4029c7ddda99ecbfa144f0afec44a192c442b6e5) (https://git.llvm.org/git/llvm 1181c40e0e24e0cca32e2609686db1f14151fc1a) (based on LLVM 7.0.332826) - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:33:21 PDT 2018
    [    0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD
    exp@exp:~$ 
    exp@exp:~$ 
    exp@exp:~$ cat main2/main2_dmesg.txt | grep "clang\|Machine"
    [    0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Flash clang version 7.0.332826 (https://git.llvm.org/git/clang 4029c7ddda99ecbfa144f0afec44a192c442b6e5) (https://git.llvm.org/git/llvm 1181c40e0e24e0cca32e2609686db1f14151fc1a) (based on LLVM 7.0.332826) - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:39:25 PDT 2018
    [    0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD
    exp@exp:~$ 
    exp@exp:~$ 
    exp@exp:~$ 
    exp@exp:~$ cat qualcomm1/qualcomm1_dmesg.txt | grep "clang\|Machine"
    [    0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Snapdragon LLVM ARM Compiler 4.0.2 for Android NDK (based on llvm.org 4.0+) - clang version 4.0.2 for Android NDK - android-ndk-r13b) #1 SMP PREEMPT Mon Jun 4 00:43:58 PDT 2018
    [    0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD
    exp@exp:~$ 
    exp@exp:~$ 
    exp@exp:~$ cat qualcomm2/qualcomm2_dmesg.txt | grep "clang\|Machine"
    [    0.000000] Linux version 3.4.113-unicornblood-hammerhead-o+ (exp@exp) (Snapdragon LLVM ARM Compiler 4.0.2 for Android NDK (based on llvm.org 4.0+) - clang version 4.0.2 for Android NDK - android-ndk-r17) #1 SMP PREEMPT Mon Jun 4 00:48:42 PDT 2018
    [    0.000000] Machine: Qualcomm MSM 8974 HAMMERHEAD (Flattened Device Tree), model: LGE MSM 8974 HAMMERHEAD
    exp@exp:~$
    [Contd.... below in next post - due to space restrictions per post]

  • [ANDROID ARM LLVM/CLANG-KERNEL ~ RESEARCH PROJECT OVERVIEW]



    1. Finding the right kernel source and kernel config that works with the Android version I had on my Nexus 5.
      • => Android(version) on my Nexus 5 is Resurrection Remix(RR) oreo with 3.4.13 hammerhead kernel(unicornblood config)
      • => Experimenting with different kernel source and hammerhead kernel config including that of AOSP

    2. Finding the most compact and quickest way to just build the kernel out of tree but using the tree's build tools
      • => Syncing the Resurrection Remix's source from its repo onto my machine
      • => Finding the make targets for building just the ramdisk instead of building the whole RR ROM which was not the goal
      • => Finding the tool and the arguments for that to build the boot.img that can be fastboot-ed

    3. Building a working kernel from source for my RR on Nexus 5, first with Android NDK r13b gcc
      • => Finding the DirtyUnicorn kernel source, building it with gcc in the first place, generating the boot.img
      • => Finding the right kernel config for the hammerhead kernel - unicornblood that's used by DirtyUnicorn repo
      • => Using the pre-built ramdisk from RR zip instead of self-built one
      • => Disabling SELinux for allowing the kernel to be fastboot-ed
      • => Debugging the ADB-over-USB not working with the DirtyUnicorn kernel image built
      • => Discussing on the Resurrection Remix forum and with the kernel developer(uname:voidz) to know the actual kernel source used in RR.

    4. Successfully booting the gcc-built kernel for RR on my Nexus 5
      • => Working kernel confimed in the first place for my ultimate Android ARM clang-kernel goal

    5. Setting up the clang-kernel build
      • => Finding the right LLVM/clang toolchain to begin with - Android NDK r13b LLVM/clang
      • => Using the binutils - assembler, linker, etc that's in Android NDK r13b

    6. Launching the clang-kernel build
      • => Disabling the options that clang doesn't recognize
      • => Examining the initial compilation erros - invalid instruction error(thrown actually by x86_64 GNU as)
      • => Fixing the assembler path to the ARM assembler
      • => Making sure it uses the right external assembler(Android EABI GNU assembler)

    7. Fixing the subsequent build errors
      • => RCU header had some static code check which had to be disabled since in clang case alone it was an error not in gcc
      • => VLAIS in various kernel components had to be changed to non-VLAIS to work with clang

    8. Updating VLAIS to non-VLAIS in various kernel components
      • => Disk encryption
      • => USB Gadget/Function Filesystem
      • => CRC32
      • => Netfilter

    9. Fixing linker errors for duplicate exception sections that clang generates
      • => Instructing the linker to leave the exceptions out while generating the final kernel image

    10. Fixing more linker errors
      • => Added missing ARM EABI memory manipulation implementations that some of the kernel code needed that the toolchain didn't provide
      • => Building successfully the kernel image with clang for the first time
    [Contd.... below in next post - due to space restrictions per post]
    1. Booting the clang-built kernel image for the first time
      • => Booting gets stuck at "Google" logo
      • => Checking whether there was any kernel panic by booting to recovery mode and checking /proc/last_kmsg
      • => Not finding anything relevant to clang-kernel boot in /proc/last_kmsg

    2. Running over a plethora of possibilities for the stuck-at-google-logo case
      • => Kernel might have not been loaded at all by bootloader for some reason
      • => Generated boot.img might have incorrect offsets for kernel, ramdisk, etc
      • => Generated kernel image might be corrupt
      • => Bootloader might be expecting gcc-specific compiler metadata instead of LLVM/clang, in kernel image header
      • => Fastboot might have some way of logging the overall boot sequence which might have give some hint

    3. Attempting to boot the clang-kernel and ramdisk with QEMU/ARM to debug like I did for x86_64 clang-kernel
      • => Booting on QEMU/ARM with both kernel and ramdisk, next with just the kernel, with GUI and without GUI
      • => Booting the kernel and ramdisk built with Android emulator, specifically QEMU/ARMel that's part of the SDK
      • => Cross-compiling Android LittleKernel bootloader for ARM and using that to boot the clang-kernel on QEMU/ARM
      • => Not seeing anything happening at all in any of the above scenarios

    4. Carrying out more debugging
      • => Turning off the Nexus 5 and doing a fastboot to get fresh /proc/last_kmsg if possible
      • => Adding various debug parameters on kernel command-line
      • => Trying out different options that fastboot has for specifying offsets, etc
      • => Researching on external hardware-based debugging like adding UART interface to get early boot logs

    5. Researching on the very first code that runs when the kernel is loaded - kernel entry point
      • => Finding start_kernel() inside init/main.c
      • => Adding some debug statements over there and not seeing them for obvious console-not-yet-initialized reason
      • => Then finding the actual kernel entry point in head.S assembly source
      • => Researching on ways to print anything at all in the ARM assembly code within head.S
      • => Finding printascii that's used for the above case but realizing it's for a serial console(UART)

    6. Understanding the ARM assembly code within head.S and its siblings and the inline documentation therein
      • => Getting to know the prerequisites of prior entering the kernel entry point in head.S

    7. Exploring the methods to confirm whether the control is indeed reaching head.S or not
      • => Checking if the LED on the bottom of Nexus 5 can be turned on/off with different colors as an indication
      • => Researching on doing something with ARM CPU like raising an exception, or a reset event as an indication
      • => Researching on different ways of restarting Nexus 5 - checking how a "reboot" command works at kernel level
      • => Looking into machine restart logic - translating that to a reset logic to be used within head.S

    8. Using reset logic for more fine-grained debugging instruction by instrucion within head.S and its siblings
      • => Noticing PC write was problematic
      • => Finding ways to branch off to destination instead of modifying PC which is not recommended as per the docs
      • => Understanding the end-to-end control flow since the kernel entry point till start_kernel() of init/main.c
      • => Following the inter-working of head.S and processor-specific assembly code during the setup
      • => Locating some control register access being problematic like that with PC

    9. Researching on other available clang/LLVM toolchains to see if they work
      • => Using Android NDK r17's LLVM/clang - not helpful - same outcome - stuck-at-google-logo case
      • => Finding main LLVM/clang source and building it to use with kernel source - not helpful - same outcome
      • => Finding Snapdragon Qualcomm LLVM/clang toolchain and using it - not helpful - same outcome

    10. Using different diff-tools to compare two binaries : clang-built kernel and gcc-built kernel
      • => hexdiff - saw some differences
      • => vimdiff - some other differenes
    [Contd.... below in next post - due to space restrictions per post]
    1. Using different binary analysis tools to examine the differences between the gcc-built and clang-built kernels
      • => Android EABI readelf - saw some ELF header information differences
      • => Android EABI objdump - compared disassembly, symbols, sections and their flags, etc

    2. Adding mechanism to use the same assembly code settings as that of gcc
      • => Using same assembler options that gcc uses while invoking the assembler for intermediate assembly code
      • => Using same assembly code setup as that of gcc for data, target architecture, etc for intermediates
      • => Automating the above so that it works for every intermediate assembly file that clang generates

    3. Reducing the optimization level of kernel build
      • => Keeping the oversmart optimization aside - O1 and Os - didn't change the stuck-at-google-logo case
      • => Disabling optimization completely - O0 - didn't work - kernel doesn't support based on what I read online

    4. Disabling the caches in kernel config as needed by head.S
      • => Updating kernel config to disable I-cache and D-cache - didn't help

    5. Trying out different assembler options
      • => Experimenting with different SP sizes, EABI versions - didn't help

    6. Trying out different clang options
      • => Using different possible stack alignments to address any incorrect assumptions around that - didn't help

    7. Correcting the SP access
      • Updating access to SP in one of the thread access kernel code as per one of the online notes

    8. Researching more on the lines on what does clang do that's not gcc doesn't
      • => Disabling all the optimizations and clang-only features if any
      • => Disabling all the intrinsic features that clang uses internally - device rebooted after a failed boot!
      • => No more stuck-at-google-logo case with the above change
      • => Witnessing the device auto-reboot with the above change - must be a kernel panic!
      • => Checking /proc/last_kmsg within recovery mode - yes, it was the clang-kernel that panic'ed - good sign!
      • => Finally, the clang-kernel has started executing after an exhaustive set of attempts - breakthrough!

    9. Locating the source of kernel panic
      • => Looking at the stacktrace revealed one of the audio codec had a buffer overrun - fixed it
      • => Rebuilding the kernel with the fix and retrying
      • => Seeing some more kernel panics - another audio codec source which had similar issue - fixed it

    10. Booting to the Android GUI with clang-kernel for the first time
      • => Fixing the kernel panics mentioned above allowed boot to move on
      • => Seeing Android animation(RR logo in my case) for the first time!
      • => Getting to the Android home screen after few seconds of wait - mission accomplished!
    [Contd.... below in next post - due to space restrictions per post]
    1. Verifying all the system information
      • => Checking kernel version to be showing LLVM/clang toolchain version, etc
      • => Examining kernel dmesg for clang specific information
      • => Checking /proc/version for the same
      • => Checking Settings/About for the same

    2. Verifying all the features work
      • => Checking Camera, Bluetooth, ADB over USB, etc
      • => Cheking WiFi - didn't work - "connected, no internet"

    3. Noticing WiFi symbol had a cross(x) symbol on it
      • => Browsing failed as expected due to no internet availability
      • => Disabling WiFi to check if cellular(LTE) network works for internet - didn't

    4. Noticing mobile network also had a cross(x) symbol on it after disabling WiFi as above
      • => Browsing with mobile network as well failed as expected due to no internet availability
      • => Verifying phone calls work - yes, they worked!

    5. Checking logcat, dmesg for any network error
      • => Noticing SELinux denials for some of the network related actions
      • => Locating the error stating bandwidth module not loaded
      • => Narrowing down to the kernel code where the possible issue is present

    6. Fixing the netfilter code for the above issue
      • => Updating one of the netfilter code with the latest code from that of the mainline kernel
      • => Rebuilt the kernel - WiFi and mobile network - both worked!

    7. Realizing all the features are now working with a clang-built ARM kernel for Android!
      • => Planning to repeat the same with the all ther remaining LLVM/clang toolchains

    8. Using Android NDK r13b's LLVM/clang in place of main LLVM/clang used so far for building the kernel
      • => Noticing Kernel panic
      • => Tracking down the kernel panic to one of the Camera MSM driver code

    9. Fixing the Camera MSM driver code in terms of device id specification
      • => Comparing with other Camera MSM driver source code and finding the difference if any
      • => Completing the device/driver id specification with the missing item - fixed the panic
      • => Booting to Android home screen this time with even the Android NDK r13's LLVM/clang-built kernel!

    10. Picking the next remaining LLVM/clang toolchains
      • => Android NDK r17's LLVM/clang - no issues in booting thus far updated kernel
      • => Qualcomm Snapdragon LLVM/clang - no issues in booting thus far updated kernel

    11. Performing round up of all the toolchains and combinations of NDK binutils
      • => Verifying all the combinations(total 8) :
        • * Android NDK r13b [gcc + binutils(as, ld, etc)]
        • * Android NDK r17 [gcc + binutils(as, ld, etc)]
        • * Android NDK r13b [LLVM/clang + binutils(as, ld, etc)]
        • * Android NDK r17 [LLVM/clang + binutils(as, ld, etc)]
        • * Main LLVM/clang + Android NDK r13b binutils(as, ld, etc)
        • * Main LLVM/clang + Android NDK r17 binutils(as, ld, etc)
        • * Snapdragon Qualcomm LLVM/clang + Android NDK r13b binutils(as, ld, etc)
        • * Snapdragon Qualcomm LLVM/clang + Android NDK r17 binutils(as, ld, etc)

      • => Confirming all of the above work - yes, worked!

    12. Automating all of the above builds and the testing of the images
      • => Facilitating automation of all the 8 combination builds
      • => Collecting the statistics - build time, image sizes, etc
      • => Summarizing the longest/shortest builds, largest/smallest zImage-dtb, largest/smallest boot.img
      • => Testing all of the images one by one for the final time

    13. Consolidating the data from the above automation
      • => Collecting the complete kernel boot(dmesg) log from each build
      • => Taking snapshots of the kernel version, Android version, build number from each build - About/System info

    14. Wrapping my Android clang-kernel research project!
      • => Done and dusted. Period.



    NOTE: For the detailed research walk-through of this project of mine, you can read more at "https://ubuntuforums.org/showthread.php?t=2394035". Look for [ANDROID ARM LLVM/CLANG-KERNEL ~ RESEARCH WALK-THROUGH] over there.

Sign In or Register to comment.