Re: x86_64 kernel with i386 userland

From: Paul H. Hargrove (PHHargrove_at_lbl_dot_gov)
Date: Wed Oct 21 2009 - 16:50:19 PDT

  • Next message: Tiantian Liu: "performance about BLCR"
    Alan,
    
    Thanks for the patch.  Sorry that I've been too busy w/ another project 
    to look at it until today.
    The same problem exists on a ppc64 (and on sparc64 if that port is ever 
    completed).
    Cross-compilation creates a chicken-and-egg with the way the current 
    configure steps are done because the setting of CR_KARCH comes too late.
    
    So, I offer the attached patch which generalizes yours and fixes the 
    cross-compilation issue by reordering some of the configure probes.
    Please let me know if the attachment works for you as well as your own 
    version does.
    
    I should note that at least in my own trials on an old ppc64 machine 
    with FC6, I did *NOT* find the kbuild magic to "just work" and therefore 
    needed to add KCC='gcc -m64' to the configure command.
    
    -Paul
    
    
    Alan Woodland wrote:
    > Hi,
    >
    > I've run into another minor problem building. When the
    > userland/toolchain is i386 it is possible still to be using an x86_64
    > kernel. If I try and configure for this kernel, e.g.:
    >
    > ./configure --with-installed-libcr --with-installed-util
    > --with-components=modules --prefix=/usr --with-linux=2.6.30-2-amd64
    >
    > Everything appears to be fine, however during building it dies:
    >
    >   CC [M]  /usr/src/modules/blcr/cr_module/kbuild/vmadump_i386.o
    > /usr/src/modules/blcr/cr_module/kbuild/vmadump_i386.c: In function
    > 'vmadump_store_cpu':
    > /usr/src/modules/blcr/cr_module/kbuild/vmadump_i386.c:56: error:
    > 'struct pt_regs' has no member named 'ds'
    > /usr/src/modules/blcr/cr_module/kbuild/vmadump_i386.c:57: error:
    > 'struct pt_regs' has no member named 'es'
    > /usr/src/modules/blcr/cr_module/kbuild/vmadump_i386.c: In function
    > 'vmadump_restore_cpu':
    > /usr/src/modules/blcr/cr_module/kbuild/vmadump_i386.c:161: error:
    > 'struct pt_regs' has no member named 'ds'
    > /usr/src/modules/blcr/cr_module/kbuild/vmadump_i386.c:162: error:
    > 'struct pt_regs' has no member named 'es'
    > make[7]: *** [/usr/src/modules/blcr/cr_module/kbuild/vmadump_i386.o] Error 1
    >
    > Presumably it shouldn't be trying to use vmadump_i386.c here...
    >
    > What's the best way to handle this? On Debian at least we can detect
    > this because either the running kernel or the target kernel will have
    > amd64 in the name, but that doesn't always have to be the case. We
    > could use file to test the kernel image itself? Or look at the length
    > of the addresses in System.map for that kernel. Or the presence of
    > startup_64 in the kernel could be tested for...
    >
    > The attached patch seems to work (counting length of kernel symbol
    > addresses). Even though it works, I can't actually test the modules it
    > produces on any x86_64 machines due to a xen related problem. The
    > patch is a little rough around the edges, but is it sane? Any
    > suggestions gratefully received!
    >
    > Kbuild magic does seems to be sorting out the c compiler to build the
    > module correctly:
    >
    > debian-devel64:/usr/src/modules/blcr/cr_module/kbuild# file *.ko
    > blcr.ko: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped
    > debian-devel64:/usr/src/modules/blcr/cr_module/kbuild# uname -a
    > Linux debian-devel64 2.6.30-1-686-bigmem #1 SMP Sat Aug 15 20:10:47
    > UTC 2009 i686 GNU/Linux
    >
    > Thanks,
    > Alan
    >   
    
    
    -- 
    Paul H. Hargrove                          PHHargrove_at_lbl_dot_gov
    Future Technologies Group                 Tel: +1-510-495-2352
    HPC Research Department                   Fax: +1-510-486-6900
    Lawrence Berkeley National Laboratory     
    
    
    Index: acinclude.m4
    ===================================================================
    RCS file: /var/local/cvs/lbnl_cr/acinclude.m4,v
    retrieving revision 1.177.14.4
    diff -u -r1.177.14.4 acinclude.m4
    --- acinclude.m4	21 Oct 2009 20:24:53 -0000	1.177.14.4
    +++ acinclude.m4	21 Oct 2009 23:47:13 -0000
    @@ -499,7 +499,7 @@
           cr_cvname="${cr_cvname} V=1"
         fi
         if test x$cross_compiling = xyes; then
    -      cr_cvname="$cr_cvname ARCH=$CR_ARCH CROSS_COMPILE=$host_alias-"
    +      cr_cvname="$cr_cvname ARCH=$CR_KARCH CROSS_COMPILE=$host_alias-"
         fi
       ])
       KBUILD_MAKE_ARGS="$cr_cvname"
    @@ -1043,8 +1043,11 @@
         CR_SET_CACHE_VAR([LINUX_VMLINUX])
         CR_CACHE_REVALIDATE([${LINUX_SYMTAB_FILE}],[ksymtab],[kernel symbol table])
       fi
    -  # Now check for consistency w/ the kernel source
    -  # XXX: Currently just check SMPness. Can this be more aggressive?
    +])
    +
    +# Now check for SYMTAB consistency w/ the kernel source
    +# XXX: Currently just check SMPness. Can this be more aggressive?
    +AC_DEFUN([CR_LINUX_SYMTAB_VALIDATE],[  
       AC_MSG_CHECKING([for SMP kernel source])
       CR_CACHED_KERNEL_COMPILE([smp_source],[
     	#include <linux/autoconf.h>
    @@ -1073,7 +1076,7 @@
     # ------------------------------------------------------
     # Helpers for CR_FIND_KSYM() and CR_FIND_EXPORTED_KSYM()
     AC_DEFUN([_CR_KSYM_INIT_PATTS],[
    -  case "$CR_ARCH" in
    +  case "$CR_KARCH" in
         ppc64)
           CR_KSYM_PATTERN_DATA=['[bBdDrRtT] ']
           CR_KSYM_PATTERN_CODE=['[dD] ']   dnl Function descriptor is data
    Index: configure.ac
    ===================================================================
    RCS file: /var/local/cvs/lbnl_cr/configure.ac,v
    retrieving revision 1.410.2.10
    diff -u -r1.410.2.10 configure.ac
    --- configure.ac	23 Aug 2009 22:58:26 -0000	1.410.2.10
    +++ configure.ac	21 Oct 2009 23:47:13 -0000
    @@ -835,14 +835,37 @@
     fi
     AC_MSG_RESULT([${CR_MODULE_DIR} ($cr_why)])
     
    +# Checks for matching (we hope) symbol table
    +CR_LINUX_SYMTAB
    +
    +# Setup CR_KARCH to enable 32-bit user-space on 64-bit kernels
    +CR_KARCH=$CR_ARCH
    +if test $cr_wordsize = 4; then
    +  # Count pointer hex digits in the System.map to id a 64-bit kernel
    +  # Based on logic provided by Alan Woodland
    +  AC_PROG_EGREP
    +  AC_MSG_CHECKING([if kernel and user-space wordsize match])
    +  if eval $LINUX_SYMTAB_CMD | head | $EGREP '^[[a-f0-9]]{16}' >/dev/null 2>&1; then
    +    AC_MSG_RESULT([no (32-bit user-space with 64-bit kernel)])
    +    case $CR_KARCH in
    +      i386)  CR_KARCH=x86_64;;
    +      ppc)   CR_KARCH=ppc64;;
    +      sparc) CR_KARCH=sparc64;;
    +    esac
    +  else
    +    AC_MSG_RESULT([yes])
    +  fi
    +fi
    +AC_SUBST(CR_KARCH)
    +
     # Configure automake/kbuild glue
     CR_CHECK_KBUILD
     
     # Configure module compile flags and perform a sanity check
     CR_SET_KCFLAGS
     
    -# Checks for matching (we hope) symbol table
    -CR_LINUX_SYMTAB
    +# Now we can check if SYMTAB matches the kernel
    +CR_LINUX_SYMTAB_VALIDATE
     
     ## Check for optional kernel headers...
     CR_CHECK_KERNEL_HEADER([linux/syscalls.h])
    @@ -1003,7 +1026,7 @@
       [mm_context_t],[unsigned long],[vdso])
     CR_CHECK_KERNEL_MEMBER([thread_info.sysenter_return],[#include <linux/thread_info.h>],
       [struct thread_info],[void *],[sysenter_return])
    -if test $CR_ARCH = arm; then
    +if test $CR_KARCH = arm; then
       CR_CHECK_KERNEL_MEMBER([thread_info.tp_value],[#include <linux/thread_info.h>],
         [struct thread_info],[unsigned long],[tp_value])
       CR_FIND_KSYM([__kuser_helper_start],[CODE],[extern int __kuser_helper_start(void);])
    @@ -1026,7 +1049,7 @@
     if test -n "${HAVE_MAP_VSYSCALL}"; then
       CR_FIND_KSYM([map_vsyscall],[CODE])
     fi
    -if test $CR_ARCH = x86_64; then
    +if test $CR_KARCH = x86_64; then
       CR_FIND_KSYM([syscall32_setup_pages],[CODE],
     	[extern int syscall32_setup_pages(void *, int);])
     fi
    @@ -1063,7 +1086,7 @@
     # which must have a prototype in asm/pgtable.h (arch-specific)
     # If we support ia64 or mips in the future we'll need to add more cases here.
     # NOTE: don't try to factor the CR_FIND_KSYM() call; it requires a literal.
    -case $CR_ARCH in
    +case $CR_KARCH in
       sparc*)
         CR_FIND_KSYM([mem_map_zero],[DATA])
         ZERO_PAGE_SYMBOL=mem_map_zero
    @@ -1094,7 +1117,7 @@
     CR_FIND_KSYM([do_sigaltstack],[CODE],
     	[extern int do_sigaltstack(const stack_t *, stack_t *, unsigned long);])
     
    -if test $CR_ARCH = i386 -o $CR_ARCH = x86_64; then
    +if test $CR_KARCH = i386 -o $CR_KARCH = x86_64; then
       CR_CHECK_KERNEL_TYPE([struct n_desc_struct],[
     	#include <linux/sched.h>
     	#include <asm/desc.h>])
    @@ -1145,7 +1168,7 @@
       fi
     fi
     
    -if test $CR_ARCH = i386; then
    +if test $CR_KARCH = i386; then
       CR_CHECK_KERNEL_MEMBER([pt_regs.xcs],[#include <asm/ptrace.h>],[struct pt_regs],[int],[xcs])
       CR_CHECK_KERNEL_MEMBER([pt_regs.xfs],[#include <asm/ptrace.h>],[struct pt_regs],[int],[xfs])
       CR_CHECK_KERNEL_MEMBER([pt_regs.xgs],[#include <asm/ptrace.h>],[struct pt_regs],[int],[xgs])
    @@ -1493,7 +1516,7 @@
     CR_FIND_KSYM([sys_mremap],[CODE],
     	[extern asmlinkage unsigned long sys_mremap(unsigned long addr, unsigned long old_len, unsigned long new_len, unsigned long flags, unsigned long new_addr);])
     CR_FIND_KSYM([do_sigaction],[CODE])
    -if test $CR_ARCH = x86_64; then
    +if test $CR_KARCH = x86_64; then
       CR_FIND_KSYM([load_gs_index],[CODE])
     
       CR_CHECK_KERNEL_CALL([read_pda],[#include <asm/percpu.h>])
    @@ -1555,7 +1578,7 @@
     fi # cr_build_modules
     
     
    -if test x"$cr_build_libcr" = xyes && expr $CR_ARCH : ppc >/dev/null; then
    +if test x"$cr_build_libcr" = xyes && expr $CR_KARCH : ppc >/dev/null; then
       AC_MSG_CHECKING([for work around for bug 2524])
       if test "$cr_build_modules,$force_bug2524" = "yes,"; then
         if test [`echo $LINUX_VER | sed -e 's/2.6.\([0-9]*\).*/\1/'`] -le 15; then
    Index: config/kbuild.mak.in
    ===================================================================
    RCS file: /var/local/cvs/lbnl_cr/config/kbuild.mak.in,v
    retrieving revision 1.24
    diff -u -r1.24 kbuild.mak.in
    --- config/kbuild.mak.in	4 Aug 2008 18:39:29 -0000	1.24
    +++ config/kbuild.mak.in	21 Oct 2009 23:47:13 -0000
    @@ -1,7 +1,7 @@
     # Here are the configuration bits needed in the "caller" to use this file
     #SUBDIR = vmadump4
     #TARGET = vmadump
    -#SOURCES = vmadump_common.c vmadump_@CR_ARCH@.c optional.h
    +#SOURCES = vmadump_common.c vmadump_@CR_KARCH@.c optional.h
     #[optional]EXTRA_CFLAGS = -I$(top_builddir)/include -I$(top_builddir) -I$(srcdir)
     #[optional]IMPORTS_FROM = /some/other/kbuild/module/dir /and/another
     #
    Index: cr_module/Makefile.am
    ===================================================================
    RCS file: /var/local/cvs/lbnl_cr/cr_module/Makefile.am,v
    retrieving revision 1.39
    diff -u -r1.39 Makefile.am
    --- cr_module/Makefile.am	3 Sep 2008 00:29:00 -0000	1.39
    +++ cr_module/Makefile.am	21 Oct 2009 23:47:13 -0000
    @@ -31,13 +31,13 @@
     
     BPROC_VERSION	= "4.0.0pre8"
     vmadump_dir	= $(top_srcdir)/vmadump4
    -vmad_sources	= $(vmadump_dir)/vmadump_common.c $(vmadump_dir)/vmadump_@CR_ARCH@.c
    +vmad_sources	= $(vmadump_dir)/vmadump_common.c $(vmadump_dir)/vmadump_@CR_KARCH@.c
     vmad_include	= -I$(vmadump_dir) -D__NR_vmadump=-1 -DBPROC_VERSION='$(BPROC_VERSION)'
     
     cr_sources	= $(my_sources) $(vmad_sources)
     INCLUDES =	-I$(top_builddir)/include	\
     		-I$(top_srcdir)/include		\
    -		-I$(srcdir)/arch/@CR_ARCH@	\
    +		-I$(srcdir)/arch/@CR_KARCH@	\
     		$(vmad_include)
     
     # echo an arbitrary make/environment variable (needed by kbuild glue)
    

  • Next message: Tiantian Liu: "performance about BLCR"