Thursday, June 27, 2019

Building Full Static libtensorflow-core.a for Linux(x86) and Android(Arm) and Running Sample App.

Building Full Static libtensorflow-core.a for Linux(x86) and Android(Arm) and Running Sample App.

LINUX
>Prerequisite

sudo apt-get install autoconf automake libtool curl make g++ unzip zlib1g-dev git python

Compilation
git clone https://github.com/tensorflow/tensorflow.git
cd tensorflow
git checkout -b r1.14
git pull
#Changes for Full Build
1.tensorflow/contrib/makefile/Makefile
 comment line
 ANDROID_TYPES := -D__ANDROID_TYPES_SLIM__
2.Remove
 -DIS_SLIM_BUILD
3.All two lines after (line 645), or search libtensorflow-core.a
+$(wildcard tensorflow/core/kernel/*.cc) \
+$(wildcard tensorflow/core/math/*.cc) \
4.In file tensorflow/core/platform/platform.h
add #define IS_MOBILE_PLATFORM at TOP.

tensorflow/contrib/makefile/build_all_linux.sh

#If there is some error in buid, comments few lines so that it do not download packages again and again  on rebuild.
+#rm -rf tensorflow/contrib/makefile/downloads
+#tensorflow/contrib/makefile/download_dependencies.sh

and rerun tensorflow/contrib/makefile/build_all_linux.sh

* if you get cc1plus oom killed ( unexpected error) reduce job count .

ANDROID
Use ndk14b, as ndk15 and later have unified header

https://android.googlesource.com/platform/ndk/+/ndk-release-r16/docs/UnifiedHeaders.md

Android build is tightly coupled with NDK version , to choose right version
see below file in tensorflow source , that will ensure your compilation
goes through.

In tensorflow/tools/ci_build/Dockerfile.android check for Variable  ANDROID_NDK_FILENAME
For tensorflow 1.14 it is android-ndk-r17c-linux-x86_64.zip
wget https://dl.google.com/android/repository/android-ndk-r14b-linux-x86_64.zip

Compilation
git clone https://github.com/tensorflow/tensorflow.git
         (git clone --recurse-submodules https://github.com/tensorflow/tensorflow.git)
 cd tensorflow
git checkout -b r1.14
git pull 
export NDK_ROOT=/opt/google/android-ndk-r14b 
tensorflow/contrib/makefile/download_dependencies.sh
          tensorflow/contrib/makefile/compile_android_protobuf.sh -c -a arm64-v8a
export HOST_NSYNC_LIB=`tensorflow/contrib/makefile/compile_nsync.sh`
export TARGET_NSYNC_LIB=`CC_PREFIX="${CC_PREFIX}" NDK_ROOT="${NDK_ROOT}" \
tensorflow/contrib/makefile/compile_nsync.sh -t android -a arm64-v8a`
make -f   tensorflow/contrib/makefile/Makefile TARGET=ANDROID ANDROID_ARCH=arm64-v8a ANDROID_TYPES="-D__ANDROID_TYPES_FULL__" 


Application Compilation on Android ( use 16b)
-----------------------------------
export NDK_ROOT=/opt/google/android-ndk-r17c
cmake -DCMAKE_TOOLCHAIN_FILE=${NDK_ROOT}/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a  -DTARGET_PLATFORM=android -DTARGET_ARCHITECTURE=arm64 -DNDK_PATH=${NDK_ROOT} ..
make


Fixing Missing Kernel Errors on Mobile

The mobile version of TensorFlow is focused on inference, and so by default the list of supported ops (defined in tensorflow/core/kernels/BUILD:android_extended_ops for Bazel and tensorflow/contrib/makefile/tf_op_files.txt for make builds) doesn't include a lot that are training related. This can cause No OpKernel was registered to support Op errors when a GraphDef is loaded, even if the op isn't going to be executed.
If you see this error and it's an op that you do actually want to run on mobile, then you'll need to make local modifications to the build files to include the right .cc file that defines it



at No comments:
Labels:

Friday, June 14, 2019

Compiling Android/linux Misc Notes on Snapdragon 820

Linux Yocto Build - 820
Linux
=====


cd /apps_proc

repo init --depth=1 -u git://codeaurora.org/platform/manifest.git -b release -m LA.HB.1.1.5-10010-8x96.0.xml --repo-url=git://codeaurora.org/tools/repo.git \
--repo-branch=caf-stable && repo sync -j64 --no-tags -c -q



Android
========


git clone --depth 1 https://chipmaster2.qti.qualcomm.com/home2/git/visteon-corporation/msm8996au-lv-0-1_hlos_dev.git

cd Linux/android

repo init -u git://codeaurora.org/quic/le/le/manifest.git -b release -m LV.HB.0.0.1-09410-8x96.0.xml --repo-url=git://codeaurora.org/tools/repo.git --\
repo-branch=caf-stable && repo sync -j64 --no-tags -c -q



/home/alok/Qualcomm/Android/qti-src/msm8996au-la-1-2_hlos_dev_boot/LINUX/android
export SHELL=/bin/bash

export AUTO_TREE_ID=msm8996
source build\envsetup.sh
lunch msm8996-userdebug


Download the Qualcomm LLVM Compiler here: https://developer.qualcomm.com/download/sdllvm/snapdragon-llvm-compiler-...

2. Extract Snapdragon-llvm-3.8.4-toolchain.tar.gz

3. Move toolchains\llvm-Snapdragon_LLVM_for_Android_3.8\prebuilt\linux-x86_64 to prebuilts/clang/linux-x86/host/sdclang-3.8 within your Android build environment.

4. Add the following lines to your device's BoardConfig:

ifneq ($(HOST_OS),darwin)

SDCLANG := true

SDCLANG_PATH := prebuilts/clang/linux-x86/host/sdclang-3.8/bin

SDCLANG_LTO_DEFS := device/qcom/common/sdllvm-lto-defs.mk

endif

5. Compile Android

==========

The Snapdragon LLVM ARM Compiler 6.0.2 can be used as a drop-in replacement
for LLVM shipped as part of the Android NDK.

For ARMv8 32-bit --> arm-linux-androideabi-4.9
For AArch64 --> aarch64-linux-android-4.9

For generating AArch64 (64-bit) code, invoke your compilation line as follows:
ndk-build NDK_TOOLCHAIN_VERSION=snapdragonclang APP_ABI="arm64-v8a" -C

8. We STRONGLY RECOMMEND the following command line flags, to be set through
APP_CFLAGS, for best performance. These options ensure that all high
performance optimization features in the Snapdragon LLVM compiler are enabled
to deliver maximum performance in 32-bit and 64-bit modes.

4





For AArch64:
/build/tools/make_standalone_toolchain_snapdragon_llvm.py \
--arch arm64 --api 28 --install-dir
The above command line specifies that you are targeting Android API level 28. If
you do not specify the API level, the default will be set to the minimum
supported level for the given architecture (currently 21 for 64-bit
architectures).

If your project does not require precise math, please set
APP_CFLAGS="-Ofast -mcpu=cortex-a57"

If your project requires IEEE 754 floating point compliance, please set
APP_CFLAGS="-O3 -mcpu=cortex-a57"


===
1. Download the Qualcomm LLVM Compiler here: https://developer.qualcomm.com/downl...llvm-compiler-...

2. Extract Snapdragon-llvm-3.8.4-toolchain.tar.gz

3. Move toolchains\llvm-Snapdragon_LLVM_for_Android_3.8\prebuilt\linux-x86_64 to prebuilts/clang/linux-x86/host/sdclang-3.8 within your Android build environment.

4. Add the following lines to your device's BoardConfig:

ifneq ($(HOST_OS),darwin)

SDCLANG := true

SDCLANG_PATH := prebuilts/clang/linux-x86/host/sdclang-3.8/bin

SDCLANG_LTO_DEFS := device/qcom/common/sdllvm-lto-defs.mk

endif

5. Compile Android


9. A standalone toolchain for the Android NDK environment using the Snapdragon
LLVM ARM compiler can be created using the make_standalone_toolchain utility.


For AArch64:
/build/tools/make_standalone_toolchain_snapdragon_llvm.py \
--arch arm64 --api 28 --install-dir
The above command line specifies that you are targeting Android API level 28. If
you do not specify the API level, the default will be set to the minimum
supported level for the given architecture (currently 21 for 64-bit
architectures).



==========




cp -r ../qti-src/msm8996au-la-1-2_hlos_dev_boot/LINUX/android/* .

Download qti-src and then run .sync.sh from inside ( caf-src)


  696  cd qti-src/msm8996au-lv-0-1_hlos_dev/
  698  cd apps_proc/
  700  mv ../../../caf-src.tar .
  702  tar xvf caf-src.tar --strip 1
  
  704  export SHELL=/bin/bash
  705  cd poky/
  706  source build/conf/set_bb_env.sh -> modify bblayers.conf ( only after souring as it is replaced)
  707  build-8x96auto-image
  708  time build-8x96auto-image
  709  source /home/alok/tools/env.sh
  710  time build-8x96auto-image

/home/alok/Qualcomm/Linux/qti-src/msm8996au-lv-0-1_hlos_dev/apps_proc/poky/build


Initialising tasks: 100% |

###################################################################################################################################################################

#######| Time: 0:00:08NOTE: Executing SetScene Tasks
WARNING: Duplicate inclusion for /home/alok/Qualcomm/Linux/qti-src/msm8996au-lv-0-1_hlos_dev/apps_proc/poky/meta-qti-bsp-

prop/conf/distro/include/security_flags.inc in /home/alok/Qualcomm/Linux/qti-src/msm8996au-lv-0-1_hlos_dev/apps_proc/poky/../meta-agl/meta-agl-


Summary: There were 134 WARNING messages shown.
/home/alok/Qualcomm/Linux/qti-src/msm8996au-lv-0-1_hlos_dev/apps_proc/poky/build
/home/alok/Qualcomm/Linux/qti-src/msm8996au-lv-0-1_hlos_dev/apps_proc/poky/build

real    209m14.173s
user    2088m56.756s
sys     271m58.120s




alok@INDFCQ4RG2:~/Qualcomm/Linux/qti-src/msm8996au-lv-0-1_hlos_dev/apps_proc/poky/build$ list-build-commands  | more

Convenience commands for building images:
   build-californium-perf-image
   build-californium-image
   build-californium-perf-debug-image
   build-californium-psm-image
   build-8009-perf-image
   build-8009-image
   build-8009-qsap-image
   build-8009-drone-image
   build-8009-drone-perf-image
   build-8017-perf-image
   build-8017-image
   build-8017-snap-image
   build-8017-qsap-image
   build-8017-qsap-perf-image
   build-9607-perf-image



build-8x96auto-sdk

alok@INDFCQ4RG2:~/Qualcomm/Linux/qti-src/msm8996au-lv-0-1_hlos_dev$ find . -name '*aarch64-toolchain*'
./apps_proc/poky/build/tmp-glibc/deploy/sdk/poky-agl-glibc-x86_64-automotive-image-aarch64-toolchain-4.99.2.target.manifest
./apps_proc/poky/build/tmp-glibc/deploy/sdk/poky-agl-glibc-x86_64-automotive-image-aarch64-toolchain-4.99.2.testdata.json
./apps_proc/poky/build/tmp-glibc/deploy/sdk/poky-agl-glibc-x86_64-automotive-image-aarch64-toolchain-4.99.2.host.manifest
./apps_proc/poky/build/tmp-glibc/deploy/sdk/poky-agl-glibc-x86_64-automotive-image-aarch64-toolchain-4.99.2.sh
./apps_proc/poky/build/tmp-glibc/work/8x96auto-agl-linux/automotive-image/1.0-r0/x86_64-deploy-automotive-image-populate-sdk/poky-agl-glibc-x86_64-automotive-

image-aarch64-toolchain-4.99.2.target.manifest
./apps_proc/poky/build/tmp-glibc/work/8x96auto-agl-linux/automotive-image/1.0-r0/x86_64-deploy-automotive-image-populate-sdk/poky-agl-glibc-x86_64-automotive-

image-aarch64-toolchain-4.99.2.testdata.json
./apps_proc/poky/build/tmp-glibc/work/8x96auto-agl-linux/automotive-image/1.0-r0/x86_64-deploy-automotive-image-populate-sdk/poky-agl-glibc-x86_64-automotive-

image-aarch64-toolchain-4.99.2.host.manifest
./apps_proc/poky/build/tmp-glibc/work/8x96auto-agl-linux/automotive-image/1.0-r0/x86_64-deploy-automotive-image-populate-sdk/poky-agl-glibc-x86_64-automotive-

image-aarch64-toolchain-4.99.2.sh



====Android Build========
  564  ls
  565  tar xvf msm8996au-la-1-2_hlos_dev_boot-1d4bfe3d24e0a10270d1b900591da4b4cd1d7c38.tar
  566  ls
  567  cd msm8996au-la-1-2_hlos_dev_boot.git/
  568  ls
  569  cd LINUX/android/
  570  ls
  571  sh sync.sh
  572  source /home/alok/tools/env.sh
  573  sh sync.sh
  574  rm -rf .repo
  575  sh sync.sh
  576  ls
  577  pwd
  578  history

root@INDFCQ4RG2:/home/alok/Qualcomm/Android/msm8996au-la-1-2_hlos_dev_boot.git/LINUX/android
export AUTO_TREE_ID=msm8996
source build/envsetup.sh
lunch msm8996-userdebug
make -j32

cd common/build
build.py --imf



=====
1. DOwnload Releaes note document
from Chipcode
RNO-180809061933-562
MSM8996AU.LV.0.1
RNO-180809061933-56299 Rev. 1 (Visteon Corporation)
September 17, 2018


2.sudo apt-get install gawk wget git-core diffstat unzip texinfo build-essential \
chrpath libsdl1.2-dev xterm subversion cvs dos2unix lzop gcc-multilib g++-multilib \
libglib2.0-dev lib32gcc1 libc6-i386 lib32z1 lib32stdc++6 \
lib32ncurses5 lib32gomp1 lib32z1-dev \
xutils-dev libtool texinfo chrpath libsdl1.2-dev automake \
flex libssl-dev texinfo chrpath python-dev


3.sudo dpkg-reconfigure dash ( Press No)

4.
sudo rm /bin/sh
sudo ln -s /bin/bash /bin/sh


5.
https://chipcode.qti.qualcomm.com and scroll down on 'Products'
section and select MSM8996AU.LV.0.1.
And click on distro that is available to you, for example, /msm8996au-lv-0-1_
hlos_dev


6. Use QFIL tool to Download if device do not goes to Fastboot Mode ( due to corrupt bootloader - in dmesg  it shows as Qualcomm Modem , windows_ USB_Bulk device)
   With FastBoot device in Dmesg it shows as google device and in windows manager as ???


7.
alok@INDFCQ4RG2:~/Qualcomm/HLOS_LV_01_16_1$ git clone --depth=1 https://chipmaster2.qti.qualcomm.com/home2/git/visteon-corporation/msm8996au-lv-0-1_hlos_dev.git
Cloning into 'msm8996au-lv-0-1_hlos_dev'...
Username for 'https://chipmaster2.qti.qualcomm.com': aprasad4@visteon.com
Password for '[https://aprasad4@visteon.com@chipmaster2.qti.qualcomm.com':India@123]https://aprasad4@visteon.com@chipmaster2.qti.qualcomm.com':India@123



Errors
=====
1.E: Package 'ia32-libs' has no installation candidate
2.lib32bz2-dev

Notes
1.modemlite for APQ8096AU
and Asic for msm8996au

2.fastboot oem select-display-panel hdmi
fastboot oem select-display-panel hdmi
or
fastboot oem select-display-panel none




Linux
=====


cd /apps_proc

repo init --depth=1 -u git://codeaurora.org/platform/manifest.git -b release -m LA.HB.1.1.5-10010-8x96.0.xml --repo-url=git://codeaurora.org/tools/repo.git \
--repo-branch=caf-stable && repo sync -j64 --no-tags -c -q



Android
========


git clone --depth 1 https://chipmaster2.qti.qualcomm.com/home2/git/visteon-corporation/msm8996au-lv-0-1_hlos_dev.git

cd Linux/android

repo init -u git://codeaurora.org/quic/le/le/manifest.git -b release -m LV.HB.0.0.1-09410-8x96.0.xml --repo-url=git://codeaurora.org/tools/repo.git --\
repo-branch=caf-stable && repo sync -j64 --no-tags -c -q



/home/alok/Qualcomm/Android/qti-src/msm8996au-la-1-2_hlos_dev_boot/LINUX/android
export SHELL=/bin/bash

export AUTO_TREE_ID=msm8996
source build\envsetup.sh
lunch msm8996-userdebug


Download the Qualcomm LLVM Compiler here: https://developer.qualcomm.com/download/sdllvm/snapdragon-llvm-compiler-...

2. Extract Snapdragon-llvm-3.8.4-toolchain.tar.gz

3. Move toolchains\llvm-Snapdragon_LLVM_for_Android_3.8\prebuilt\linux-x86_64 to prebuilts/clang/linux-x86/host/sdclang-3.8 within your Android build environment.

4. Add the following lines to your device's BoardConfig:

ifneq ($(HOST_OS),darwin)

SDCLANG := true

SDCLANG_PATH := prebuilts/clang/linux-x86/host/sdclang-3.8/bin

SDCLANG_LTO_DEFS := device/qcom/common/sdllvm-lto-defs.mk

endif

5. Compile Android

==========

The Snapdragon LLVM ARM Compiler 6.0.2 can be used as a drop-in replacement
for LLVM shipped as part of the Android NDK.

For ARMv8 32-bit --> arm-linux-androideabi-4.9
For AArch64 --> aarch64-linux-android-4.9

For generating AArch64 (64-bit) code, invoke your compilation line as follows:
ndk-build NDK_TOOLCHAIN_VERSION=snapdragonclang APP_ABI="arm64-v8a" -C

8. We STRONGLY RECOMMEND the following command line flags, to be set through
APP_CFLAGS, for best performance. These options ensure that all high
performance optimization features in the Snapdragon LLVM compiler are enabled
to deliver maximum performance in 32-bit and 64-bit modes.

4





For AArch64:
/build/tools/make_standalone_toolchain_snapdragon_llvm.py \
--arch arm64 --api 28 --install-dir
The above command line specifies that you are targeting Android API level 28. If
you do not specify the API level, the default will be set to the minimum
supported level for the given architecture (currently 21 for 64-bit
architectures).

If your project does not require precise math, please set
APP_CFLAGS="-Ofast -mcpu=cortex-a57"

If your project requires IEEE 754 floating point compliance, please set
APP_CFLAGS="-O3 -mcpu=cortex-a57"



Flashing Android  for QFIL
=========================

cp LINUX/android/out/target/product/msm8996/boot.img /test/LINUX/android/out/target/product/msm8996/.
cp LINUX/android/out/target/product/msm8996/system.img /test/LINUX/android/out/target/product/msm8996/.
cp LINUX/android/out/target/product/msm8996/vendor.img /test/LINUX/android/out/target/product/msm8996/.
cp LINUX/android/out/target/product/msm8996/persist.img /test/LINUX/android/out/target/product/msm8996/.
cp LINUX/android/out/target/product/msm8996/userdata.img /test/LINUX/android/out/target/product/msm8996/.
cp LINUX/android/out/target/product/msm8996/emmc_appsboot.mbn /test/LINUX/android/out/target/product/msm8996/.
cp LINUX/android/out/target/product/msm8996/obj/EMMC_BOOTLOADER_OBJ/build-msm8996/lk /test/LINUX/android/out/target/product/msm8996/obj/EMMC_BOOTLOADER_OBJ/build-msm8996/.
cp LINUX/android/out/target/product/msm8996/obj/KERNEL_OBJ/vmlinuz  /test/LINUX/android/out/target/product/msm8996/obj/KERNEL_OBJ/.

LINUX/android/vendor/qcom/proprietary/usb/host/windows/prebuilt/adb.exe
LINUX/android/vendor/qcom/proprietary/usb/host/windows/prebuilt/AdbWinApi.dll
LINUX/android/vendor/qcom/proprietary/usb/host/windows/prebuilt/fastboot.exe


mkdir -p /test/LINUX/android/out/target/product/msm8996/obj/EMMC_BOOTLOADER_OBJ/build-msm8996/
mkdir -p /test/LINUX/android/out/target/product/msm8996/obj/KERNEL_OBJ/



root@INDFCQ4RG2:/test# tar cvf LINUX.tar LINUX/*
LINUX/android/
LINUX/android/out/
LINUX/android/out/target/
LINUX/android/out/target/product/
LINUX/android/out/target/product/msm8996/
LINUX/android/out/target/product/msm8996/obj/
LINUX/android/out/target/product/msm8996/obj/EMMC_BOOTLOADER_OBJ/
LINUX/android/out/target/product/msm8996/obj/EMMC_BOOTLOADER_OBJ/build-msm8996/
LINUX/android/out/target/product/msm8996/obj/EMMC_BOOTLOADER_OBJ/build-msm8996/lk
LINUX/android/out/target/product/msm8996/obj/KERNEL_OBJ/
LINUX/android/out/target/product/msm8996/obj/KERNEL_OBJ/vmlinux
LINUX/android/out/target/product/msm8996/system.img
LINUX/android/out/target/product/msm8996/userdata.img
LINUX/android/out/target/product/msm8996/persist.img
LINUX/android/out/target/product/msm8996/boot.img
LINUX/android/out/target/product/msm8996/vendor.img
LINUX/android/out/target/product/msm8996/emmc_appsboot.mbn



Install
root@INDFCQ4RG2:/home/alok/Qualcomm/Android# cd /pkg/qct/software/llvm/release/arm/3.5.2.1/
aarch64-linux-gnu/    armv5-none-eabi/      armv7-linux-gnueabi/  armv7-nofp-none-eabi/ bin/                  libexec/              RELEASE_NOTES         tools/
aarch64-none-elf/     armv6m-none-eabi/     armv7m-none-eabi/     armv7-none-eabi/      lib/                  NOTICE                share/


1.      Install snapdragon LLVM  ( snapdragon_sd_llvm_arm.lnx.4.0_installer_41100.109-23-18_05_37_56.zip)
 /pkg/qct/software/llvm/release/arm/3.5.2.1/ ( this path is must)
2.      Download and extract msm8996au-la-1-2_hlos_dev_boot ( remember this build takes about 200GB)






at No comments:
Subscribe to: Posts (Atom)

Featured Post

XDP - Getting Started with XDP (Linux)

  • SIP/VOIP/IMS Interview Questions
    Below is the list of VOIP Interview questions , that will cover most of the interview questions If you find it useful please do write comme...
  • XDP - Getting Started with XDP (Linux)
      XDP Introduced in Linux 4.8 eBPF hook at the driver level (ingress) Intercept packet before it reaches the stack, before allocating sk_buf...
  • Fake i2c device( i2c-stub)
    If you want to test I2C client implementation without actual hardware you can do this with i2c-stub. Kernel has a Fake I2C master a...