Bug 216537

Summary: [cmake][armv7] OFFLINE_ASM_BACKEND is wrongly set building in a native ARMv7 env in the top of a ARM64 host
Product: WebKit Reporter: Pablo Saavedra <psaavedra>
Component: CMakeAssignee: Pablo Saavedra <psaavedra>
Status: RESOLVED WORKSFORME    
Severity: Normal CC: annulen, bburg, ews-watchlist, gyuyoung.kim, hi, keith_miller, mark.lam, msaboff, ross.kirsling, ryuan.choi, saam, sergio, tzagallo, ysuzuki
Priority: P2    
Version: WebKit Nightly Build   
Hardware: Unspecified   
OS: Unspecified   
Attachments:
Description Flags
patch
none
patch
none
patch mark.lam: review-

Description Pablo Saavedra 2020-09-15 07:06:04 PDT 
The OFFLINE_ASM_BACKEND cmake var is wrongly defined when you try to build WebKit in a chroom ARMv/armhf chroot (or similar) in  real ARM64 host. This bad definition of the OFFLINE_ASM_BACKEND var leads in a build error:


```
[899/4520] cd /webkit-debian-nightly/build/wpewebkit/obj-arm-linux-gnueabihf/Source/JavaScriptCore && /usr/bin/ruby /webkit-debian-nightly/build/wpewebkit/Source/JavaScriptCore/offlineasm/generate_offset_extractor.rb -I/webkit-debian-nightly/build/wpewebkit/obj-arm-linux-gnueabihf/DerivedSources/JavaScriptCore/ /webkit-debian-nightly/build/wpewebkit/Source/JavaScriptCore/llint/LowLevelInterpreter.asm /webkit-debian-nightly/build/wpewebkit/obj-arm-linux-gnueabihf/bin/LLIntSettingsExtractor /webkit-debian-nightly/build/wpewebkit/obj-arm-linux-gnueabihf/DerivedSources/JavaScriptCore/LLIntDesiredOffsets.h ARM64
FAILED: DerivedSources/JavaScriptCore/LLIntDesiredOffsets.h 
```



How to replicate the enviroment:


* Install Debian for ARM64 in the server.

* Using a powerful ARM64 server like this:
  * RAPTOR EV-883832-X3-0001
  * CPU: 32 Physical Cores @ 3.3 GHz (1 X AMPERE EMAG)
  * Memory: 128 GB of DDR4 ECC RAM
  * ref: https://store.avantek.co.uk/ampere-emag-64bit-arm-workstation.html

* Install Debian Stable for ARM64 in the server.
* Create an chroot environment with debootstrap for ARMv7. (ARMv7 can be executed by a ARM64 kernel)
* Enter en the chroot an set up the build as usual


The Debian enviroment into the chroot is ARMv7/armhf

```
root@c299a99a3ff5:/webkit-debian-nightly/build/wpewebkit# dpkg-architecture  
DEB_BUILD_ARCH=armhf
DEB_BUILD_ARCH_ABI=eabihf
DEB_BUILD_ARCH_BITS=32
DEB_BUILD_ARCH_CPU=arm
DEB_BUILD_ARCH_ENDIAN=little
DEB_BUILD_ARCH_LIBC=gnu
DEB_BUILD_ARCH_OS=linux
DEB_BUILD_GNU_CPU=arm
DEB_BUILD_GNU_SYSTEM=linux-gnueabihf
DEB_BUILD_GNU_TYPE=arm-linux-gnueabihf
DEB_BUILD_MULTIARCH=arm-linux-gnueabihf
DEB_HOST_ARCH=armhf
DEB_HOST_ARCH_ABI=eabihf
DEB_HOST_ARCH_BITS=32
DEB_HOST_ARCH_CPU=arm
DEB_HOST_ARCH_ENDIAN=little
DEB_HOST_ARCH_LIBC=gnu
DEB_HOST_ARCH_OS=linux
DEB_HOST_GNU_CPU=arm
DEB_HOST_GNU_SYSTEM=linux-gnueabihf
DEB_HOST_GNU_TYPE=arm-linux-gnueabihf
DEB_HOST_MULTIARCH=arm-linux-gnueabihf
DEB_TARGET_ARCH=armhf
DEB_TARGET_ARCH_ABI=eabihf
DEB_TARGET_ARCH_BITS=32
DEB_TARGET_ARCH_CPU=arm
DEB_TARGET_ARCH_ENDIAN=little
DEB_TARGET_ARCH_LIBC=gnu
DEB_TARGET_ARCH_OS=linux
DEB_TARGET_GNU_CPU=arm
DEB_TARGET_GNU_SYSTEM=linux-gnueabihf
DEB_TARGET_GNU_TYPE=arm-linux-gnueabihf
DEB_TARGET_MULTIARCH=arm-linux-gnueabihf
```

But the kernel is ARM 64 bits:


root@c299a99a3ff5:/webkit-debian-nightly/build/wpewebkit# uname.old  -a
Linux c299a99a3ff5 5.6.0-0.bpo.2-arm64 #1 SMP Debian 5.6.14-2~bpo10+1 (2020-06-09) aarch64 GNU/Linux
root@c299a99a3ff5:/webkit-debian-nightly/build/wpewebkit# uname.old  -m
aarch64



CMake uses `uname -m` to check and define the CMAKE_HOST_SYSTEM_PROCESSOR:

```
/usr/share/cmake-3.16/Modules/CMakeDetermineSystem.cmake

        exec_program(${CMAKE_UNAME} ARGS -m OUTPUT_VARIABLE CMAKE_HOST_SYSTEM_PROCESSOR
```

what is used by WebKit's LOWERCASE_CMAKE_SYSTEM_PROCESSOR function to check the system:


```
 if (LOWERCASE_CMAKE_SYSTEM_PROCESSOR MATCHES "(^aarch64|^arm64)")
    set(WTF_CPU_ARM64 1)
elseif (LOWERCASE_CMAKE_SYSTEM_PROCESSOR MATCHES "^arm")
    set(WTF_CPU_ARM 1)
```
Comment 1 Pablo Saavedra 2020-09-15 07:08:38 PDT 
Created attachment 408815 [details]
patch
Comment 2 Pablo Saavedra 2020-09-15 07:09:33 PDT 
Created attachment 408816 [details]
patch
Comment 3 Pablo Saavedra 2020-09-15 07:13:01 PDT 
Created attachment 408817 [details]
patch
Comment 4 Pablo Saavedra 2020-09-15 07:18:45 PDT 
Comment on attachment 408817 [details]
patch

View in context: https://bugs.webkit.org/attachment.cgi?id=408817&action=review

> Source/JavaScriptCore/CMakeLists.txt:255
> +    elseif (WTF_CPU_ARM64)

A pure ARM64 will not have Thumb2 extensions defined. Only an ARMv7 system (using a or not a ARM64 kernel) could expose those extensions.

> Source/cmake/OptionsCommon.cmake:23
> +if (WTF_CPU_ARM OR WTF_CPU_ARM64)

The WTF_CPU_ARM64 is defined based on the `uname -m`. The kernel still could be a ARM64 but running a ARM 32bits basesystem.
Comment 5 Mark Lam 2020-09-15 07:19:10 PDT 
Comment on attachment 408817 [details]
patch

This patch doesn’t make sense to me.  If you’re cross compiling for armv7 on a arm64 host, why is your cross compiler environment defining WTF_CPU_ARM64?  Can you explain that?  I suspect that that is where the real issue is.
Comment 6 Pablo Saavedra 2020-09-15 07:21:43 PDT 
(In reply to Mark Lam from comment #5)
> Comment on attachment 408817 [details]
> patch
> 
> This patch doesn’t make sense to me.  If you’re cross compiling for armv7 on
> a arm64 host, why is your cross compiler environment defining WTF_CPU_ARM64?
> Can you explain that?  I suspect that that is where the real issue is.

I'm not crosscompiling. The kernel is arm64 but the basesystem is armhf. The compilation is done purely natively.
Comment 7 Pablo Saavedra 2020-09-15 07:28:22 PDT 
(In reply to Pablo Saavedra from comment #6)
> (In reply to Mark Lam from comment #5)
> > Comment on attachment 408817 [details]
> > patch
> > 
> > This patch doesn’t make sense to me.  If you’re cross compiling for armv7 on
> > a arm64 host, why is your cross compiler environment defining WTF_CPU_ARM64?
> > Can you explain that?  I suspect that that is where the real issue is.
> 
> I'm not crosscompiling. The kernel is arm64 but the basesystem is armhf. The
> compilation is done purely natively.

* HOST Kernel (ARM 64 bits) (ARM64 / AARCH64 / Linux 5.6.0-0.bpo.2-arm64 #1 SMP Debian 5.6.14-2~bpo10+1 (2020-06-09) aarch64 GNU/Linux)
* HOST Basesystem libs (ARM 32 bits) (Debian armhf / ARMv7 Thumb2 Hard Floating point)
* TARGET (ARM 32 bits): The same above ^
Comment 8 Mark Lam 2020-09-15 07:33:57 PDT 
Comment on attachment 408817 [details]
patch

View in context: https://bugs.webkit.org/attachment.cgi?id=408817&action=review

r- for now

>> Source/cmake/OptionsCommon.cmake:23
>> +if (WTF_CPU_ARM OR WTF_CPU_ARM64)
> 
> The WTF_CPU_ARM64 is defined based on the `uname -m`. The kernel still could be a ARM64 but running a ARM 32bits basesystem.

I think this is the wrong approach.  I think that if you’re cross compiling for 32-bit ARM, WTF_CPU_ARM64 shouldn’t be defined.  Is there a reason that this cannot be?
Comment 9 Mark Lam 2020-09-15 08:19:26 PDT 
(In reply to Pablo Saavedra from comment #7)
> (In reply to Pablo Saavedra from comment #6)
> > (In reply to Mark Lam from comment #5)
> > > Comment on attachment 408817 [details]
> > > patch
> > > 
> > > This patch doesn’t make sense to me.  If you’re cross compiling for armv7 on
> > > a arm64 host, why is your cross compiler environment defining WTF_CPU_ARM64?
> > > Can you explain that?  I suspect that that is where the real issue is.
> > 
> > I'm not crosscompiling. The kernel is arm64 but the basesystem is armhf. The
> > compilation is done purely natively.
> 
> * HOST Kernel (ARM 64 bits) (ARM64 / AARCH64 / Linux 5.6.0-0.bpo.2-arm64 #1
> SMP Debian 5.6.14-2~bpo10+1 (2020-06-09) aarch64 GNU/Linux)
> * HOST Basesystem libs (ARM 32 bits) (Debian armhf / ARMv7 Thumb2 Hard
> Floating point)
> * TARGET (ARM 32 bits): The same above ^

Same thing.  WebKit is not set up to have such ambiguity.  Even if you get past the offlineasm here, you’ll run into all manner of problems later simply because ARM64 is defined.

In your case, it looks like you want to build and run a 32-bit arm binary on a 64 bit host.  This is effectively cross compiling and the solution you need is the same as that used for cross compilation i.e. the target platform code needs to detect that you’re targeting 32-bit ARM and not pollute it with the detail that it will end up running on a 64-bit host.  That detail is orthogonal to how it needs to build.
Comment 10 Pablo Saavedra 2020-09-15 08:32:12 PDT 
(In reply to Mark Lam from comment #9)
> (In reply to Pablo Saavedra from comment #7)
> > (In reply to Pablo Saavedra from comment #6)
> > > (In reply to Mark Lam from comment #5)
> > > > Comment on attachment 408817 [details]
> > > > patch
> > > > 
> > > > This patch doesn’t make sense to me.  If you’re cross compiling for armv7 on
> > > > a arm64 host, why is your cross compiler environment defining WTF_CPU_ARM64?
> > > > Can you explain that?  I suspect that that is where the real issue is.
> > > 
> > > I'm not crosscompiling. The kernel is arm64 but the basesystem is armhf. The
> > > compilation is done purely natively.
> > 
> > * HOST Kernel (ARM 64 bits) (ARM64 / AARCH64 / Linux 5.6.0-0.bpo.2-arm64 #1
> > SMP Debian 5.6.14-2~bpo10+1 (2020-06-09) aarch64 GNU/Linux)
> > * HOST Basesystem libs (ARM 32 bits) (Debian armhf / ARMv7 Thumb2 Hard
> > Floating point)
> > * TARGET (ARM 32 bits): The same above ^
> 
> Same thing.  WebKit is not set up to have such ambiguity.  Even if you get
> past the offlineasm here, you’ll run into all manner of problems later
> simply because ARM64 is defined.
> 
> In your case, it looks like you want to build and run a 32-bit arm binary on
> a 64 bit host.  This is effectively cross compiling and the solution you
> need is the same as that used for cross compilation i.e. the target platform
> code needs to detect that you’re targeting 32-bit ARM and not pollute it
> with the detail that it will end up running on a 64-bit host.  That detail
> is orthogonal to how it needs to build.

I see your point. 

Lately I was sharing my thoughts with aperez@igalia.com. He told about `setarch` 


setarch  -  change  reported  architecture  in  new program environment
            and/or set personality flags
 


```
# uname -m
aarch64

# setarch --list
uname26
linux32
linux64
armv7l
armv8l
armh
arm
arm64
aarch64

# setarch arm

# uname -m
armv8l <<<
```


I suppose I can use this in my environment and keep it the WK's CMakes as it is.