[pve-devel] Please help: beginner trying to compile Proxmox kernel 4.15

Discussion:

The Anh Tran

2018-09-10 03:12:27 UTC

Dear Proxmox Dev Team,

I'm trying to re-compile Proxmox VE 5.2 kernel, 4.15 branch, but no
success so far.

I've followed this README:

https://git.proxmox.com/?p=pve-common.git;a=blob_plain;f=README.dev;hb=HEAD

###############################

Install a fresh Proxmox VE 5.2 on my machine

Download Debian 9.5 (standard) container template from machine-ip:8006 \
local \ Content screen. Spawn a container instance from this template,
120GB root disk.

SSH to container instance

apt-get install all build prerequisites per README instruction:

/apt-get install git nano screen patch fakeroot build-essential
devscripts libncurses5 libncurses5-dev libssl-dev bc flex bison
libelf-dev libaudit-dev libgtk2.0-dev libperl-dev libperl-dev asciidoc
xmlto gnupg gnupg2 rsync lintian debhelper libiberty-dev/

###############################

/mkdir /source/

/cd /source/

############ Build common PVE perl ###############

/git clone git://git.proxmox.com/git/pve-common.git/

/cd /source/pve-common/

/make/

/dpkg -i libpve-common-perl_5.0-38_all.deb/

############# Pull pve-kernel ################

/git clone -b pve-kernel-4.15 --single-branch
git://git.proxmox.com/git/pve-kernel.git/

############## Pull zfsonlinux ##########

/cd /source/pve-kernel/submodules/

/git clone git://git.proxmox.com/git/zfsonlinux.git/

########### Pull ubuntu-bionic ###########

/git clone git://git.proxmox.com/git/mirror_ubuntu-bionic-kernel.git/

/mv mirror_ubuntu-bionic-kernel ubuntu-bionic/

####### Build it ##########

/cd /source/pve-kernel/

/make 2>&1 | tee ~/build-4.15.1.log/

Build fail with this message:

/install: target
'debian/pve-kernel-4.15.18-4-pve/lib/modules/4.15.18-4-pve/kernel/drivers/net/ethernet/intel/igb/'
is not a directory: No such file or directory//
//debian/rules:99: recipe for target '.install_mark' failed/

Inside this directory:
//source/pve-kernel-4.15/build/debian/pve-kernel-4.15.18-4-pve/lib/modules/

There are:

4.15.18-4-pve     <=    this dir is empty

4.15.18-4-pve+ <=    this dir has files/
/

I edit this file /source/pve-kernel/debian/rules, add 2 commands at line 106

/        ${MAKE} -C ${KERNEL_SRC}
INSTALL_MOD_PATH=${BUILD_DIR}/debian/${PVE_KERNEL_PKG}/ modules_install//
//
/*/        rmdir debian/${PVE_KERNEL_PKG}/lib/modules/${KVNAME}/**/
/**/        cp -a debian/${PVE_KERNEL_PKG}/lib/modules/${KVNAME}+
debian/${PVE_KERNEL_PKG}/lib/modules/${KVNAME}/**/
/*/
//        ## install latest ibg driver//
/

########## make again ############

/cd /source/pve-kernel/

/make 2>&1 | tee ~/build-4.15.2.log/

Build fail at this error:
/input file
'debian/pve-headers-4.15.18-4-pve/usr/src/linux-headers-4.15.18-4-pve/Module.symvers'
does not exist//
//debian/rules:245: recipe for target 'abi-4.15.18-4-pve' failed/

There is no linux-headers-4.15.18-4-pve directory inside.

And there is no linux header to download from Proxmox git site.

=> don't know how to fix this error.

Any input is appreciated.

Best regards.

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Thomas Lamprecht

2018-09-10 05:50:09 UTC

Permalink

Hi,

On 9/10/18 5:12 AM, The Anh Tran wrote:> Dear Proxmox Dev Team,

I'm trying to re-compile Proxmox VE 5.2 kernel, 4.15 branch, but no success so far.
https://git.proxmox.com/?p=pve-common.git;a=blob_plain;f=README.dev;hb=HEAD
###############################
Install a fresh Proxmox VE 5.2 on my machine
Download Debian 9.5 (standard) container template from machine-ip:8006 \ local \ Content screen. Spawn a container instance from this template, 120GB root disk.
SSH to container instance
/apt-get install git nano screen patch fakeroot build-essential devscripts libncurses5 libncurses5-dev libssl-dev bc flex bison libelf-dev libaudit-dev libgtk2.0-dev libperl-dev libperl-dev asciidoc xmlto gnupg gnupg2 rsync lintian debhelper libiberty-dev/
###############################
/mkdir /source/
/cd /source/
############ Build common PVE perl ###############
/git clone git://git.proxmox.com/git/pve-common.git/
/cd /source/pve-common/
/make/
/dpkg -i libpve-common-perl_5.0-38_all.deb/

FYI, pve-common not needed for building the kernel only.
It's the common library for all of our perl code (so all but kernel, qemu, lxc - to name the most popular ones)

############# Pull pve-kernel ################
/git clone -b pve-kernel-4.15 --single-branch git://git.proxmox.com/git/pve-kernel.git/
############## Pull zfsonlinux ##########
/cd /source/pve-kernel/submodules/
/git clone git://git.proxmox.com/git/zfsonlinux.git/

FYI, zfsonlinux also has two submodules which may need initializing.

cd /source/pve-kernel/submodules/zfsonlinux && make submodule

also ensure that the correct commit is checked out on all submodules, i.e.,
git status -uno

Does not shows changes (especially on submodules), if so use:

git submodule update --init

to checkout the correct commit.

########### Pull ubuntu-bionic ###########
/git clone git://git.proxmox.com/git/mirror_ubuntu-bionic-kernel.git/
/mv mirror_ubuntu-bionic-kernel ubuntu-bionic/
####### Build it ##########
/cd /source/pve-kernel/
/make 2>&1 | tee ~/build-4.15.1.log/
/install: target 'debian/pve-kernel-4.15.18-4-pve/lib/modules/4.15.18-4-pve/kernel/drivers/net/ethernet/intel/igb/' is not a directory: No such file or directory//
//debian/rules:99: recipe for target '.install_mark' failed/

this is just a symptom of an earlier error, something did not go right earlier.
can you please post your whole build log?
It's probably to big for the mailing list, so maybe use https://gist.github.com/
(or something other, somewhat trustworthy to click on)

Inside this directory: //source/pve-kernel-4.15/build/debian/pve-kernel-4.15.18-4-pve/lib/modules/
4.15.18-4-pve <= this dir is empty
4.15.18-4-pve+ <= this dir has files/
/
I edit this file /source/pve-kernel/debian/rules, add 2 commands at line 106
/ ${MAKE} -C ${KERNEL_SRC} INSTALL_MOD_PATH=${BUILD_DIR}/debian/${PVE_KERNEL_PKG}/ modules_install//
//
/*/ rmdir debian/${PVE_KERNEL_PKG}/lib/modules/${KVNAME}/**/
/**/ cp -a debian/${PVE_KERNEL_PKG}/lib/modules/${KVNAME}+ debian/${PVE_KERNEL_PKG}/lib/modules/${KVNAME}/**/
/*/
// ## install latest ibg driver//
/
########## make again ############
/cd /source/pve-kernel/
/make 2>&1 | tee ~/build-4.15.2.log/
/input file 'debian/pve-headers-4.15.18-4-pve/usr/src/linux-headers-4.15.18-4-pve/Module.symvers' does not exist//
//debian/rules:245: recipe for target 'abi-4.15.18-4-pve' failed/
There is no linux-headers-4.15.18-4-pve directory inside.
And there is no linux header to download from Proxmox git site.

this gets generated by the package you want to build, it's just the header files from the kernel...

cheers,
Thomas

=> don't know how to fix this error.
Any input is appreciated.
Best regards.

The Anh Tran

2018-09-10 11:55:02 UTC

Permalink

Dear Thomas,

Post by Thomas Lamprecht
Hi,
FYI, zfsonlinux also has two submodules which may need initializing.
cd /source/pve-kernel/submodules/zfsonlinux && make submodule
also ensure that the correct commit is checked out on all submodules, i.e.,
git status -uno
git submodule update --init
to checkout the correct commit.

submodules / zfsonlinux is on latest commit:

/compile:~# cd /source/pve-kernel/submodules/zfsonlinux///
//compile:/source/pve-kernel/submodules/zfsonlinux# git status -uno//
//On branch master//
//Your branch is up-to-date with 'origin/master'.//
//nothing to commit (use -u to show untracked files)//
//compile:/source/pve-kernel/submodules/zfsonlinux# make submodule//
//test -f "zfs-debian/debian/changelog" || git submodule update --init//
//test -f "spl-debian/debian/changelog" || git submodule update --init//

/I try "make submodule", and "make all" on zfsonlinux, it builds .deb
just fine.

Do you mean i have to clone these git:
mirror_spl-debian.git
mirror_zfs-debian.git
into zfsonlinux directory?

Post by Thomas Lamprecht
/install: target
'debian/pve-kernel-4.15.18-4-pve/lib/modules/4.15.18-4-pve/kernel/drivers/net/ethernet/intel/igb/'
is not a directory: No such file or directory//

Post by The Anh Tran
//debian/rules:99: recipe for target '.install_mark' failed/

Fresh rebuild still fail at the above step.

In previous mail, i have attached 2 build run's logs. To reduce size to
fit 80KB limit, repetitive outputs from kernel build script has been
removed.

And i also upload to my company hosting:
https://cloud.tgdd.vn/index.php/s/d6JajWDF7nfmJuh

And google drive link:
https://drive.google.com/drive/folders/1YJmu--aOtPBj0iqj_LXKp5mx4lTvih9_?usp=sharing

Post by Thomas Lamprecht
4.15.18-4-pve <= this dir is empty
4.15.18-4-pve+ <= this dir has files/
/
I edit this file /source/pve-kernel/debian/rules, add 2 commands at line 106
/ ${MAKE} -C ${KERNEL_SRC} INSTALL_MOD_PATH=${BUILD_DIR}/debian/${PVE_KERNEL_PKG}/ modules_install//
//
/*/ rmdir debian/${PVE_KERNEL_PKG}/lib/modules/${KVNAME}/**/
/**/ cp -a debian/${PVE_KERNEL_PKG}/lib/modules/${KVNAME}+ debian/${PVE_KERNEL_PKG}/lib/modules/${KVNAME}/**/
/*/
// ## install latest ibg driver//
/
########## make again ############
/cd /source/pve-kernel/
/make 2>&1 | tee ~/build-4.15.2.log/
/input file 'debian/pve-headers-4.15.18-4-pve/usr/src/linux-headers-4.15.18-4-pve/Module.symvers' does not exist//
//debian/rules:245: recipe for target 'abi-4.15.18-4-pve' failed/
There is no linux-headers-4.15.18-4-pve directory inside.
And there is no linux header to download from Proxmox git site.
this gets generated by the package you want to build, it's just the header files from the kernel...
cheers,
Thomas

Best regards.

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Fabian Grünbichler

2018-09-10 20:53:55 UTC

Permalink

Post by The Anh Tran
[...]

/install: target 'debian/pve-kernel-4.15.18-4-pve/lib/modules/4.15.18-4-pve/kernel/drivers/net/ethernet/intel/igb/'
is not a directory: No such file or directory//

Post by The Anh Tran
//debian/rules:99: recipe for target '.install_mark' failed/

Fresh rebuild still fail at the above step.

I am not sure whether your manual git clones messed anything up or
whether you somehow manage to trigger some error in our Make logic
(entirely possible!), but a plain "make" after

"git clone -b pve-kernel-4.15 git://git.proxmox.com/git/pve-kernel.git"

should and does work (here), as long as you have all the needed
build-dependencies installed. note that libpve-common-perl is actually
required for one of the helper scripts, but not (yet) contained in
debian/control.in (patch incoming sometime this week, unless somebody
beats me to it).

Post by The Anh Tran
In previous mail, i have attached 2 build run's logs. To reduce size to fit
80KB limit, repetitive outputs from kernel build script has been removed.
https://cloud.tgdd.vn/index.php/s/d6JajWDF7nfmJuh
https://drive.google.com/drive/folders/1YJmu--aOtPBj0iqj_LXKp5mx4lTvih9_?usp=sharing

4.15.18-4-pve <= this dir is empty
4.15.18-4-pve+ <= this dir has files/

maybe you accidentally added the '+' somewhere while editing/viewing a
file (like the Makefile, debian/rules, the generated file with all the
environment variables, ...)? the path should not contain a '+'!

check an excerpt of a build log here:

LD [M] drivers/net/ethernet/intel/e1000/e1000.ko
LD [M] drivers/net/ethernet/intel/e1000e/e1000e.ko
LD [M] drivers/net/ethernet/intel/fm10k/fm10k.ko
LD [M] drivers/net/ethernet/intel/i40e/i40e.ko
LD [M] drivers/net/ethernet/intel/i40evf/i40evf.ko
LD [M] drivers/net/ethernet/intel/igb/igb.ko
LD [M] drivers/net/ethernet/intel/igbvf/igbvf.ko
LD [M] drivers/net/ethernet/intel/ixgb/ixgb.ko
LD [M] drivers/net/ethernet/intel/ixgbe/ixgbe.ko
LD [M] drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko
LD [M] drivers/net/ethernet/jme.ko
--
CC util/dso.o
CC /rpool/kerneltest/pve-kernel/build/modules/igb-5.3.5.18/src/igb.mod.o
CC util/symbol.o
CC util/symbol_fprintf.o
CC tests/mmap-basic.o
LD [M] /rpool/kerneltest/pve-kernel/build/modules/igb-5.3.5.18/src/igb.ko
make[3]: Leaving directory '/rpool/kerneltest/pve-kernel/build/ubuntu-bionic'
make[2]: Leaving directory '/rpool/kerneltest/pve-kernel/build/modules/igb-5.3.5.18/src'
cp modules/igb-5.3.5.18/src/igb.ko modules/
CC util/color.o
CC tests/perf-record.o
yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
--
INSTALL drivers/net/ethernet/intel/e1000/e1000.ko
INSTALL drivers/net/ethernet/intel/e1000e/e1000e.ko
INSTALL drivers/net/ethernet/intel/fm10k/fm10k.ko
INSTALL drivers/net/ethernet/intel/i40e/i40e.ko
INSTALL drivers/net/ethernet/intel/i40evf/i40evf.ko
INSTALL drivers/net/ethernet/intel/igb/igb.ko
INSTALL drivers/net/ethernet/intel/igbvf/igbvf.ko
INSTALL drivers/net/ethernet/intel/ixgb/ixgb.ko
INSTALL drivers/net/ethernet/intel/ixgbe/ixgbe.ko
INSTALL drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko
INSTALL drivers/net/ethernet/jme.ko
--
INSTALL ubuntu/xr-usb-serial/xr_usb_serial_common.ko
INSTALL virt/lib/irqbypass.ko
DEPMOD 4.15.18-4-pve
make[2]: Leaving directory '/rpool/kerneltest/pve-kernel/build/ubuntu-bionic'
## install latest ibg driver
install -m 644 modules/igb.ko debian/pve-kernel-4.15.18-4-pve/lib/modules/4.15.18-4-pve/kernel/drivers/net/ethernet/intel/igb/
# install latest e1000e driver
install -m 644 modules/e1000e.ko debian/pve-kernel-4.15.18-4-pve/lib/modules/4.15.18-4-pve/kernel/drivers/net/ethernet/intel/e1000e/
# install zfs drivers
install -d -m 0755 debian/pve-kernel-4.15.18-4-pve/lib/modules/4.15.18-4-pve/zfs
install -m 644 modules/spl.ko modules/splat.ko modules/zfs.ko modules/zavl.ko modules/znvpair.ko modules/zunicode.ko modules/zcommon.ko modules/zpios.ko modules/icp.ko debian/pve-kernel-4.15.18-4-pve/lib/modules/4.15.18-4-pve/zfs

no 4.15.18-4-pve+ anywhere in the log, as expected.

what does "git status" output in your local working copy of the repo?

The Anh Tran

2018-09-11 09:54:46 UTC

Permalink

Dear Teams,

Post by Fabian GrÃ¼nbichler
I am not sure whether your manual git clones messed anything up or
whether you somehow manage to trigger some error in our Make logic
(entirely possible!), but a plain "make" after
"git clone -b pve-kernel-4.15 git://git.proxmox.com/git/pve-kernel.git"
should and does work (here), as long as you have all the needed
build-dependencies installed. note that libpve-common-perl is actually
required for one of the helper scripts, but not (yet) contained in
debian/control.in (patch incoming sometime this week, unless somebody
beats me to it).
maybe you accidentally added the '+' somewhere while editing/viewing a
file (like the Makefile, debian/rules, the generated file with all the
environment variables, ...)? the path should not contain a '+'!
LD [M] drivers/net/ethernet/intel/e1000/e1000.ko
kernel-4.15.18-4-pve/lib/modules/4.15.18-4-pve/zfs

...........remove for clarity ...........

Post by Fabian GrÃ¼nbichler
no 4.15.18-4-pve+ anywhere in the log, as expected.
what does "git status" output in your local working copy of the repo?

Today i download newest Debian 9 template, 9.5-1, spawn a new container.

Re-run build script again, pull latest from git.

/compile-pve:/source# cd mirror_ubuntu-bionic-kernel///
//compile-pve:/source/mirror_ubuntu-bionic-kernel# git status -uno//
//On branch master//
//Your branch is up-to-date with 'origin/master'.//
//nothing to commit (use -u to show untracked files)//
/

/compile-pve:/source/zfsonlinux# git status -uno/

/On branch master//
//Your branch is up-to-date with 'origin/master'.//
//nothing to commit (use -u to show untracked files)//
/

/compile-pve:/source# git clone -b pve-kernel-4.15
git://git.proxmox.com/git/pve-kernel.git//
/

/compile-pve:/source# cd pve-kernel/

/compile-pve:/source/pve-kernel#
/

/cd ../pve-kernel/git status -uno//
//On branch pve-kernel-4.15//
//Your branch is up-to-date with 'origin/pve-kernel-4.15'.//
//nothing to commit (use -u to show untracked files)//
/

Make still error at "wrong directory" message

File *builld/ubuntu-bionic/include/config/kernel.release* has this content:

4.15.18-4-pve+

Most of .o and .ko files also have "-pve+" in their version string.

I'm looking for which script generate that "-pve+" suffix.

I've uploaded putty log to google drive:

https://drive.google.com/open?id=1YJmu--aOtPBj0iqj_LXKp5mx4lTvih9_

Best regards.

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Fabian Grünbichler

2018-09-12 09:28:50 UTC

Permalink

Post by The Anh Tran
Dear Teams,

...........remove for clarity ...........

Post by Fabian GrÃ¼nbichler
no 4.15.18-4-pve+ anywhere in the log, as expected.
what does "git status" output in your local working copy of the repo?

which is not what you are supposed to do. just clone pve-kernel:

$ git clone -b pve-kernel-4.15 git://git.proxmox.com/git/pve-kernel.git

and run make

$ cd pve-kernel; make

that will pull in all the needed submodules automatically, no need for
manually cloning and copying stuff around. note that if I haven't
misread your "log", you didn't install libpve-common-perl, which means
the build will fail at a later stage anyway because of this missing
depednency.

Post by The Anh Tran
Make still error at "wrong directory" message
4.15.18-4-pve+
Most of .o and .ko files also have "-pve+" in their version string.
I'm looking for which script generate that "-pve+" suffix.

nothing in our scripts adds a + there. please check the following gist:

https://gist.github.com/Fabian-Gruenbichler/f427703810508c3dab345459d32ecb8a

and compare the log and contents of build/debian/rules.d/env.mk with
yours.

Thomas Lamprecht

2018-09-12 12:00:04 UTC

Permalink

Post by Fabian GrÃ¼nbichler

Post by The Anh Tran
/On branch master//
//Your branch is up-to-date with 'origin/master'.//
//nothing to commit (use -u to show untracked files)//
/
/compile-pve:/source# git clone -b pve-kernel-4.15
git://git.proxmox.com/git/pve-kernel.git//
/
/compile-pve:/source# cd pve-kernel/
/compile-pve:/source/pve-kernel#
/
/cd ../pve-kernel/git status -uno//
//On branch pve-kernel-4.15//
//Your branch is up-to-date with 'origin/pve-kernel-4.15'.//
//nothing to commit (use -u to show untracked files)//
/

$ git clone -b pve-kernel-4.15 git://git.proxmox.com/git/pve-kernel.git
and run make
$ cd pve-kernel; make
that will pull in all the needed submodules automatically, no need for
manually cloning and copying stuff around. note that if I haven't
misread your "log", you didn't install libpve-common-perl, which means
the build will fail at a later stage anyway because of this missing
depednency.

The plus comes from the fact that your kernel directory is not clean!

This (upstream) kernel helper scripts adds it:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/setlocalversion#n165

The solution is not to tinker around with the script but:
1) build as Fabian directed you, i.e., a single clone plus make - no tinkering
2) if you add changes on top, do so in a clean way.

I can just setup a clean Debian install the build dependencies and build just
fine.

The Anh Tran

2018-09-13 09:08:03 UTC

Permalink

Dear Teams,

I've followed Fabian's instruction, and build process is success.

It turns out that: submodules/ubuntu-bionic downloading by build script,
is different than git clone
git://git.proxmox.com/git/mirror_ubuntu-bionic-kernel.git

Minor suggestion: please update this tutorial
https://forum.proxmox.com/threads/tutorial-compile-proxmox-ve-5-with-patched-intel-iommu-driver-to-remove-rmrr-check.36374/

I'm testing my modification on this source.

Thank you for your support.
Best regards.

Post by Thomas Lamprecht

Post by Fabian GrÃ¼nbichler

The plus comes from the fact that your kernel directory is not clean!
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/setlocalversion#n165
1) build as Fabian directed you, i.e., a single clone plus make - no tinkering
2) if you add changes on top, do so in a clean way.
I can just setup a clean Debian install the build dependencies and build just
fine.
_______________________________________________
pve-devel mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Fabian Grünbichler

2018-09-13 11:13:33 UTC

Permalink

Post by The Anh Tran
Dear Teams,
I've followed Fabian's instruction, and build process is success.

that's a good start ;)

Post by The Anh Tran
It turns out that: submodules/ubuntu-bionic downloading by build script, is
different than git clone
git://git.proxmox.com/git/mirror_ubuntu-bionic-kernel.git

no it's not, modulo some different tracking in .git under the hood.

Post by The Anh Tran
Minor suggestion: please update this tutorial
https://forum.proxmox.com/threads/tutorial-compile-proxmox-ve-5-with-patched-intel-iommu-driver-to-remove-rmrr-check.36374/
I'm testing my modification on this source.

that tutorial is not written by us ;) but it also does not at any point
say to copy separately cloned git repositories into sub directories of
the local clone of pve-kernel.git . the purpose of cloning the ubuntu
bionic mirror into a separate checkout is to have a clear separation for
working on your changes - you just need to put the final patch into your
copy of pve-kernel.git, not manually copy the whole modified kernel
sources.

I suggest we follow up in the forum thread instead of here on pve-devel.

The Anh Tran

2018-09-22 05:19:02 UTC

Permalink

Dear Teams,

I have Dell EMC ScaleIO (renamed to VxFlexOS) license.

ScaleIO has 3 components:
SDS: server storage provider. This component could be installed on
Proxmox ok.
MDM: distributed storage metadata management. This is installed on a VM
for HA & live-migration requirement.
SDC: storage client consumer. This component could be installed on
Proxmox ok.

I have posted a SR ticket on EMC site. They replied: "unable to support
Proxmox distribution due to ... reasons".
=> have to solve this myself.

Sdc has a kernel module: scini.ko, which is a pre-compiled binary by
EMC, for each support OS version (Redhat, Ubuntu, ...).
scini.ko, which compiled on ubuntu xenial 16.04, has this module version
magic string: "4.15.0-33-generic SMP mod_unload".
Promox kernel 4.15.18-xy, while based on Ubuntu hwe kernel 4.15.18, has
module magic string: "4.15.18-xy-pve SMP mod_unload modversions"

SDC boot script /etc/init.d/scini run this: "insmod /bin/emc/scini.ko"
Magic string mismatch => kernel refuse to load
I could change to force load: "insmod -f /bin/emc/scini.ko"
But Proxmox compile flag inside debian/rules doesn't have
"CONFIG_MODULE_FORCE_LOAD"

So, i could make scini.ko run on Proxmox by re-compile kernel, with some
changes:
_ add "-e CONFIG_MODULE_FORCE_LOAD \" in pve-kernel/debian/rules
_ or modify pve-kernel/debian/rules: "-d CONFIG_MODVERSIONS \", and edit
Makefile version string vars, from "4.15.18-xy-pve" to "4.15.0-xy-generic".

May I know the reason why you add "CONFIG_MODVERSIONS" or not enable
"CONFIG_MODULE_FORCE_LOAD" flag?
Best regards.

Post by Fabian GrÃ¼nbichler

Post by The Anh Tran
Dear Teams,
I've followed Fabian's instruction, and build process is success.

that's a good start ;)

Post by The Anh Tran
It turns out that: submodules/ubuntu-bionic downloading by build script, is
different than git clone
git://git.proxmox.com/git/mirror_ubuntu-bionic-kernel.git

no it's not, modulo some different tracking in .git under the hood.

that tutorial is not written by us ;) but it also does not at any point
say to copy separately cloned git repositories into sub directories of
the local clone of pve-kernel.git . the purpose of cloning the ubuntu
bionic mirror into a separate checkout is to have a clear separation for
working on your changes - you just need to put the final patch into your
copy of pve-kernel.git, not manually copy the whole modified kernel
sources.
I suggest we follow up in the forum thread instead of here on pve-devel.
_______________________________________________
pve-devel mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Fabian Grünbichler

2018-09-24 16:09:43 UTC

Permalink

Post by The Anh Tran
Dear Teams,
I have Dell EMC ScaleIO (renamed to VxFlexOS) license.
SDS: server storage provider. This component could be installed on Proxmox
ok.
MDM: distributed storage metadata management. This is installed on a VM for
HA & live-migration requirement.
SDC: storage client consumer. This component could be installed on Proxmox
ok.
I have posted a SR ticket on EMC site. They replied: "unable to support
Proxmox distribution due to ... reasons".
=> have to solve this myself.
Sdc has a kernel module: scini.ko, which is a pre-compiled binary by EMC,
for each support OS version (Redhat, Ubuntu, ...).
scini.ko, which compiled on ubuntu xenial 16.04, has this module version
magic string: "4.15.0-33-generic SMP mod_unload".
Promox kernel 4.15.18-xy, while based on Ubuntu hwe kernel 4.15.18, has
module magic string: "4.15.18-xy-pve SMP mod_unload modversions"

if you don't have a source for that module, (reliably) using it will not
be possible unless the vendor compiles it for you for every PVE kernel
update that changes the ABI.

Post by The Anh Tran
SDC boot script /etc/init.d/scini run this: "insmod /bin/emc/scini.ko"
Magic string mismatch => kernel refuse to load
I could change to force load: "insmod -f /bin/emc/scini.ko"
But Proxmox compile flag inside debian/rules doesn't have
"CONFIG_MODULE_FORCE_LOAD"
So, i could make scini.ko run on Proxmox by re-compile kernel, with some
_ add "-e CONFIG_MODULE_FORCE_LOAD \" in pve-kernel/debian/rules
_ or modify pve-kernel/debian/rules: "-d CONFIG_MODVERSIONS \", and edit
Makefile version string vars, from "4.15.18-xy-pve" to "4.15.0-xy-generic".
May I know the reason why you add "CONFIG_MODVERSIONS" or not enable
"CONFIG_MODULE_FORCE_LOAD" flag?

even if we would enable force loading, there is absolutely no guarantee
that the module will actually work - (kernel) ABIs exist for a reason,
and our kernel and Ubuntu's are very close, but not ABI-compatible.

The Anh Tran

2018-09-25 08:52:30 UTC

Permalink

Dear Fabian,

Post by Fabian GrÃ¼nbichler
if you don't have a source for that module, (reliably) using it will not
be possible unless the vendor compiles it for you for every PVE kernel
update that changes the ABI.
even if we would enable force loading, there is absolutely no guarantee
that the module will actually work - (kernel) ABIs exist for a reason,
and our kernel and Ubuntu's are very close, but not ABI-compatible.
_______________________________________________
pve-devel mailing list
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

I'm stress-testing scini.ko on force-load-enabled kernel now:
add/remove volume, change vol size, connect/disconnect devices, migrate
VMs ...
while top VMs issue heavy random readwrite IOPS.

Before make run, i did git-checkout revision which as close as possible
to Ubuntu's release:
Ubuntu Git: 4.15.0-33.36~16.04.1 Ubuntu-hwe-4.15.0-33.36_16.04.1 4.15.18
Proxmox Git: 2018-08-24 bump version to 4.15.18-22
https://git.proxmox.com/?p=pve-kernel.git;a=commit;h=2b76f08c16c9e48fce96c8d5d7346d8dd2b3fe4a

Doing a diff on 2 kernel's abi files, and on 2 kernel's config files, i
read that there are major differences on NVME, BRIDGE_NETFILTER being
moved from modules to internal.
Other things seem the same.

Hope this sdc works well, better than nothing works.

Thank you for spending your time support me.
Best regards.

---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus