Re: [OctDev] rpm spec questions for octave-forge

[David Bateman]

Michael Goffioul wrote:
> On 5/20/07, dbateman <address@hidden> wrote:
>   
>> A -local/-global option can't really work as it causes issues with the
>> local_list/global_list options needed for sandboxing pkg installs for
>> testing and temporary RPM builds..
>>
>> A good option might be "(geteuid() == 0)" which should always work on Unix
>> systems and on Windows where geteuid doesn't exist it will return true as
>> well...
>>     
>
> I would still then provide a way to force "local" installation for cases
> where the user does not have write access to Octave installation
> directory under Win32 (what you propose above always force global
> installation under Win32).
>
> Michael.
>   
The problem here is how to treat the sandboxing I use in the "make
check" code in octave-forge and the temporary builds in the RPM. For
this I manipulate the prefix, global_list and local_list variables to
force the installation locations. If we add a "-local" flag, it only
makes sense if this flag forces the prefix to be something else, and then

pkg prefix /some/strange/path
pkg install -local weird-package-1.0.0.tar.gz

will have an unexpected result.

Thinking about it, maybe we can have a persistent flag that indicates
whether the user has set the prefix and in that case don't change the
prefix. Humm, ok then in this case it does make sense..

Consider the attached patch that also adds the ability to do

pkg rebuild -auto weird-package

to change the status of the autoloading for a particular package or

pkg rebuild -auto

to flag all packages as autoloaded.. The patch also attempts to get the
status of autoloading from the presence or absence of packinfo/.autoload
rather than the DESCRIPTION file without the -auto/-noauto flags.

D.

-- 
David Bateman                                address@hidden
Motorola Labs - Paris                        +33 1 69 35 48 04 (Ph) 
Parc Les Algorithmes, Commune de St Aubin    +33 6 72 01 06 33 (Mob) 
91193 Gif-Sur-Yvette FRANCE                  +33 1 69 35 77 01 (Fax) 

The information contained in this communication has been classified as: 

[x] General Business Information 
[ ] Motorola Internal Use Only 
[ ] Motorola Confidential Proprietary

*** ./scripts/miscellaneous/copyfile.m.orig42   2007-03-08 20:40:01.000000000 
+0100
--- ./scripts/miscellaneous/copyfile.m  2007-05-20 21:45:07.215214574 +0200
***************
*** 40,46 ****
    ## FIXME -- maybe use the same method as in ls to allow users control
    ## over the command that is executed.
  
!   if (ispc () && ! isunix () && isempty (file_in_path (EXEC_PATH, "cp")))
      ## Windows.
      cmd = "cmd /C xcopy /E";
      cmd_force_flag = "/Y";
--- 40,46 ----
    ## FIXME -- maybe use the same method as in ls to allow users control
    ## over the command that is executed.
  
!   if (ispc () && ! isunix () && isempty (file_in_path (EXEC_PATH, "cp.exe")))
      ## Windows.
      cmd = "cmd /C xcopy /E";
      cmd_force_flag = "/Y";
***************
*** 73,78 ****
--- 73,83 ----
      f1 = sprintf ("\"%s\" ", f1{:});
  
      f2 = tilde_expand (f2);
+   
+     if (ispc () && ! isunix () && ! isempty (file_in_path (EXEC_PATH, 
"cp.exe")))
+       f1 = strrep (f1, "\\", "/");
+       f2 = strrep (f2, "\\", "/");
+     endif
  
      ## Copy the files.
      [err, msg] = system (sprintf ("%s %s \"%s\"", cmd, f1, f2));
*** ./scripts/pkg/pkg.m.orig42  2007-05-18 23:38:13.000000000 +0200
--- ./scripts/pkg/pkg.m 2007-05-21 16:33:55.308924886 +0200
***************
*** 32,52 ****
  ## @noindent
  ## installs the package found in the file @code{image-1.0.0.tar.gz}.
  ##
! ## If @var{option} is @code{-nodeps} the package manager will disable the
! ## dependency checking. That way it is possible to install a package even
! ## if it depends on another package that's not installed on the system.
! ## @strong{Use this option with care.}
  ##
! ## If @var{option} is @code{-noauto} the package manager will not
! ## automatically load the installed package when starting Octave,
! ## even if the package requests that it is.
  ##
! ## If @var{option} is @code{-auto} the package manager will
! ## automatically load the installed package when starting Octave,
! ## even if the package requests that it isn't.
  ##
- ## Finally, if @var{option} is @code{-verbose} the package manager will
- ## print the output of all of the commands that are performed
  ## @item uninstall
  ## Uninstall named packages.  For example,
  ## @example
--- 32,66 ----
  ## @noindent
  ## installs the package found in the file @code{image-1.0.0.tar.gz}.
  ##
! ## The @var{option} variable can contain options that affect the manner
! ## in which a package is installed. These options can be one or more of
  ##
! ## @table @code
! ## @item -nodeps
! ## The package manager will disable the dependency checking. That way it 
! ## is possible to install a package even if it depends on another package 
! ## that's not installed on the system. @strong{Use this option with care.}
  ##
! ## @item -noauto
! ## The package manager will not automatically load the installed package 
! ## when starting Octave, even if the package requests that it is.
! ##
! ## @item -auto
! ## The package manager will automatically load the installed package when 
! ## starting Octave, even if the package requests that it isn't.
! ##
! ## @item -local
! ## A local installation is forced, even if the user has system privileges.
! ##
! ## @item -global
! ## A global installation is forced, even if the user doesn't normally have
! ## system privileges
! ##
! ## @item -verbose
! ## The package manager will print the output of all of the commands that are 
! ## performed.
! ## @end table
  ##
  ## @item uninstall
  ## Uninstall named packages.  For example,
  ## @example
***************
*** 124,129 ****
--- 138,154 ----
  ## @example
  ## pkg global_list
  ## @end example
+ ## @item rebuild
+ ## Rebuilds the package database from the installed directories. This can 
+ ## be used in cases where for some reason the package database is corrupted.
+ ## It can also take the @code{-auto} and @code{-noauto} options to allow the
+ ## autolaoding state of a package to be changed. For example
+ ##
+ ## @example
+ ## pkg rebuild -noauto image
+ ## @end example
+ ##
+ ## will remove the autoloading status of the image package.
  ## @end table
  ## @end deftypefn
  
***************
*** 132,151 ****
  
  function [local_packages, global_packages] = pkg (varargin)
    ## Installation prefix (XXX: what should these be on windows?)
    persistent prefix = -1;
    persistent local_list = tilde_expand (fullfile("~", ".octave_packages"));
    persistent global_list = fullfile (OCTAVE_HOME (), "share", "octave",
                                     "octave_packages");
    mlock ();
  
    if (prefix == -1)
!     if (issuperuser ())
        prefix = fullfile (OCTAVE_HOME (), "share", "octave", "packages");
      else
        prefix = fullfile ("~", "octave");
      endif
    endif
-   prefix = tilde_expand (prefix);
  
    ## Handle input
    if (length (varargin) == 0 || ! iscellstr (varargin))
--- 157,178 ----
  
  function [local_packages, global_packages] = pkg (varargin)
    ## Installation prefix (XXX: what should these be on windows?)
+   persistent user_prefix = false;
    persistent prefix = -1;
    persistent local_list = tilde_expand (fullfile("~", ".octave_packages"));
    persistent global_list = fullfile (OCTAVE_HOME (), "share", "octave",
                                     "octave_packages");
    mlock ();
  
+   global_install = issuperuser ();
    if (prefix == -1)
!     if (global_install)
        prefix = fullfile (OCTAVE_HOME (), "share", "octave", "packages");
      else
        prefix = fullfile ("~", "octave");
      endif
+     prefix = tilde_expand (prefix);
    endif
  
    ## Handle input
    if (length (varargin) == 0 || ! iscellstr (varargin))
***************
*** 166,173 ****
        auto = 1;
        case "-verbose"
        verbose = true;
        case {"list", "install", "uninstall", "load", "unload", ...
!           "prefix", "local_list", "global_list"}
        action = varargin{i};
        otherwise
        files{end+1} = varargin{i};
--- 193,210 ----
        auto = 1;
        case "-verbose"
        verbose = true;
+       case "-local"
+       global_install = false;
+       if (! user_prefix)
+         prefix = fullfile ("~", "octave");
+       endif
+       case "-global"
+       global_install = true;
+       if (! user_prefix)
+         prefix = fullfile (OCTAVE_HOME (), "share", "octave", "packages");
+       endif
        case {"list", "install", "uninstall", "load", "unload", ...
!           "prefix", "local_list", "global_list", "rebuild"}
        action = varargin{i};
        otherwise
        files{end+1} = varargin{i};
***************
*** 192,204 ****
        if (length (files) == 0)
        error ("you must specify at least one filename when calling 'pkg 
install'");
        endif
!       install (files, deps, auto, prefix, verbose, local_list, global_list);
  
      case "uninstall"
        if (length (files) == 0)
        error ("you must specify at least one package when calling 'pkg 
uninstall'");
        endif
!       uninstall (files, deps, verbose, local_list, global_list);
  
      case "load"
        if (length (files) == 0)
--- 229,243 ----
        if (length (files) == 0)
        error ("you must specify at least one filename when calling 'pkg 
install'");
        endif
!       install (files, deps, auto, prefix, verbose, local_list, 
!              global_list, global_install);
  
      case "uninstall"
        if (length (files) == 0)
        error ("you must specify at least one package when calling 'pkg 
uninstall'");
        endif
!       uninstall (files, deps, verbose, local_list, 
!                global_list, global_install);
  
      case "load"
        if (length (files) == 0)
***************
*** 222,227 ****
--- 261,268 ----
        ## if (!strcmp(prefix(end), filesep))
        ##   prefix(end+1) = filesep;
        ## endif
+       prefix = tilde_expand (prefix);
+       user_prefix = true;
        else
        error ("you must specify a prefix directory, or request an output 
argument");
        endif
***************
*** 247,257 ****
--- 288,381 ----
        else
        error ("you must specify a global_list file, or request an output 
argument");
        endif
+ 
+     case "rebuild"
+       if (global_install)
+       global_packages = rebuild (prefix, global_list, auto, verbose);
+       save (global_list, "global_packages");
+       local_packages = global_packages;
+       else
+       local_packages = rebuild (prefix, local_list, files, auto, verbose);
+       save (local_list, "local_packages");
+       endif
+ 
      otherwise
        error ("you must specify a valid action for 'pkg'. See 'help pkg' for 
details");
    endswitch
  endfunction
  
+ function descriptions = rebuild (prefix, list, files, auto, verbose)
+   if (isempty (files))
+     [dirlist, err, msg] = readdir (prefix);
+     if (err)
+       error ("couldn't read directory %s: %s", prefix, msg);
+     endif
+     ## the two first entries of dirlist are "." and ".."
+     dirlist([1,2]) = [];
+   else
+     old_descriptions = installed_packages (list, list);
+     wd = pwd ();
+     cd (prefix);
+     dirlist = glob (cellfun(@(x) [x, '-*'], files, 'UniformOutput', 0))
+     cd (wd);
+   endif
+   descriptions = {};
+   for k = 1:length (dirlist)
+     descfile = fullfile (prefix, dirlist{k}, "packinfo", "DESCRIPTION");
+     if (verbose)
+       printf ("recreating package description from %s\n", dirlist{k});
+     endif
+     if (exist (descfile, "file"))
+       desc = get_description (descfile);
+       desc.dir = fullfile (prefix, dirlist{k});
+       if (auto != 0)
+       if (exist (fullfile (desc.dir, "packinfo", ".autoload"), "file"))
+         unlink (fullfile (desc.dir, "packinfo", ".autoload"));
+       endif
+         if (auto < 0)
+         desc.autoload = 0;
+       elseif (auto > 0)
+         desc.autoload = 1;
+         fclose (fopen (fullfile (desc.dir, "packinfo", ".autoload"), "wt"));
+       endif
+       else
+       if (exist (fullfile (desc.dir, "packinfo", ".autoload"), "file"))
+         desc.autoload = 1;
+       else
+         desc.autoload = 0;
+       endif
+       endif
+       descriptions{end + 1} = desc;
+     elseif (verbose)
+       warning ("directory %s is not a valid package", dirlist{k});
+     endif
+   endfor
+ 
+   if (isempty (files))
+     ## We are rebuilding for a particular package(s) so we should take
+     ## care to keep the other untouched packages in the descriptions
+     descriptions = {desriptions{:}, old_desriptions{:}};
+ 
+     dup = [];
+     for i = 1:length (descriptions)
+       if (find (dup, i))
+       continue;
+       endif
+       for j = (i+1):length (descriptions)
+       if (find (dup, j))
+         continue;
+       endif
+       if (strcmp (descriptions{i}.name, descriptions{j}.name))
+         dup = [dup, j];
+       endif
+       endfor
+     endfor
+     if (! isempty (dup))
+       descriptions (dup) = [];
+     endif  
+   endif
+ endfunction
+ 
  function auto = isautoload (desc)
  auto = false;
  if (isfield (desc{1}, "autoload"))
***************
*** 266,273 ****
  endif
  endfunction
  
! function install (files, handle_deps, autoload, prefix, verbose, local_list, 
global_list)
!   global_install = issuperuser ();
  
    # Check that the directory in prefix exist. If it doesn't: create it!
    if (! exist (prefix, "dir"))
--- 390,396 ----
  endif
  endfunction
  
! function install (files, handle_deps, autoload, prefix, verbose, local_list, 
global_list, global_install)
  
    # Check that the directory in prefix exist. If it doesn't: create it!
    if (! exist (prefix, "dir"))
***************
*** 299,364 ****
      for i = 1:length (files)
        tgz = files{i};
  
!       ## Create a temporary directory 
!       tmpdir = tmpnam ();
!       tmpdirs{end+1} = tmpdir;
!       if (verbose)
!       printf ("mkdir (%s)\n", tmpdir);
!       endif
!       [status, msg] = mkdir (tmpdir);
!       if (status != 1)
!       error ("couldn't create temporary directory: %s", msg);
!       endif
! 
!       ## Uncompress the package
!       if (verbose)
!       printf ("untar (%s, %s)\n", tgz, tmpdir);
!       endif
!       untar (tgz, tmpdir);
! 
!       ## Get the name of the directories produced by tar
!       [dirlist, err, msg] = readdir (tmpdir);
!       if (err)
!       error ("couldn't read directory produced by tar: %s", msg);
!       endif
! 
!       if (length (dirlist) > 3)
!       error ("bundles of packages are not allowed")
!       endif
! 
!       ## the two first entries of dirlist are "." and ".."
!       for k = 3:length (dirlist)
!       packdir = fullfile (tmpdir, dirlist{k});
!       packdirs{end+1} = packdir;
  
!       ## Make sure the package contains necessary files
!       verify_directory (packdir);
  
!       ## Read the DESCRIPTION file
!       filename = fullfile (packdir, "DESCRIPTION");
!       desc = get_description (filename);
  
!       ## Verify that package name corresponds with filename
!       [dummy, nm] = fileparts (tgz); 
!       if ((length (nm) >= length (desc.name))
!           && ! strcmp (desc.name, nm(1:length(desc.name))))
!         error ("package name '%s' doesn't correspond to its filename '%s'", 
desc.name, nm);
        endif
  
!       ## Set default installation directory
!       desc.dir = fullfile (prefix, strcat (desc.name, "-", desc.version));
  
!       ## Save desc
!       descriptions{end+1} = desc;
  
!       ## Are any of the new packages already installed?
!       ## If so we'll remove the old version.
!       for j = 1:length (packages)
!         if (strcmp (packages{j}.name, desc.name))
!           packages_to_uninstall(end+1) = j;
!         endif
!       endfor
!       endfor        
      endfor
    catch
      ## Something went wrong, delete tmpdirs
--- 422,491 ----
      for i = 1:length (files)
        tgz = files{i};
  
!       if (exist (tgz, "file"))
!       ## Create a temporary directory 
!       tmpdir = tmpnam ();
!       tmpdirs{end+1} = tmpdir;
!         if (verbose)
!         printf ("mkdir (%s)\n", tmpdir);
!       endif
!       [status, msg] = mkdir (tmpdir);
!       if (status != 1)
!         error ("couldn't create temporary directory: %s", msg);
!       endif
  
!       ## Uncompress the package
!       if (verbose)
!         printf ("untar (%s, %s)\n", tgz, tmpdir);
!       endif
!       untar (tgz, tmpdir);
  
!       ## Get the name of the directories produced by tar
!       [dirlist, err, msg] = readdir (tmpdir);
!       if (err)
!         error ("couldn't read directory produced by tar: %s", msg);
!       endif
  
!       if (length (dirlist) > 3)
!         error ("bundles of packages are not allowed")
        endif
  
!       ## the two first entries of dirlist are "." and ".."
!       for k = 3:length (dirlist)
!         packdir = fullfile (tmpdir, dirlist{k});
!         packdirs{end+1} = packdir;
! 
!         ## Make sure the package contains necessary files
!         verify_directory (packdir);
! 
!         ## Read the DESCRIPTION file
!         filename = fullfile (packdir, "DESCRIPTION");
!         desc = get_description (filename);
! 
!         ## Verify that package name corresponds with filename
!         [dummy, nm] = fileparts (tgz); 
!         if ((length (nm) >= length (desc.name))
!             && ! strcmp (desc.name, nm(1:length(desc.name))))
!           error ("package name '%s' doesn't correspond to its filename '%s'", 
desc.name, nm);
!         endif
  
!         ## Set default installation directory
!         desc.dir = fullfile (prefix, strcat (desc.name, "-", desc.version));
  
!         ## Save desc
!         descriptions{end+1} = desc;
! 
!         ## Are any of the new packages already installed?
!         ## If so we'll remove the old version.
!         for j = 1:length (packages)
!           if (strcmp (packages{j}.name, desc.name))
!             packages_to_uninstall(end+1) = j;
!           endif
!         endfor
!       endfor        
!       else
!       warning ("file %s does not exist", tgz);
!       endif
      endfor
    catch
      ## Something went wrong, delete tmpdirs
***************
*** 416,422 ****
    try
      for i = packages_to_uninstall
        uninstall ({installed_packages{i}.name}, false, verbose, local_list, 
!                global_list);
      endfor
    catch
      ## Something went wrong, delete tmpdirs
--- 543,549 ----
    try
      for i = packages_to_uninstall
        uninstall ({installed_packages{i}.name}, false, verbose, local_list, 
!                global_list, global_install);
      endfor
    catch
      ## Something went wrong, delete tmpdirs
***************
*** 514,524 ****
    endif
  endfunction
  
! function uninstall (pkgnames, handle_deps, verbose, local_list, global_list)
    ## Get the list of installed packages
    [local_packages, global_packages] = installed_packages(local_list, 
                                                         global_list);
!   if (issuperuser ())
      installed_packages = {local_packages{:}, global_packages{:}};
    else
      installed_packages = local_packages;
--- 641,652 ----
    endif
  endfunction
  
! function uninstall (pkgnames, handle_deps, verbose, local_list, 
!                   global_list, global_install)
    ## Get the list of installed packages
    [local_packages, global_packages] = installed_packages(local_list, 
                                                         global_list);
!   if (global_install)
      installed_packages = {local_packages{:}, global_packages{:}};
    else
      installed_packages = local_packages;
***************
*** 535,541 ****
  
    ## Are all the packages that should be uninstalled already installed?
    if (length (delete_idx) != length (pkgnames))
!     if (issuperuser ())
        ## Try again for a locally installed package
        installed_packages = local_packages
  
--- 663,669 ----
  
    ## Are all the packages that should be uninstalled already installed?
    if (length (delete_idx) != length (pkgnames))
!     if (global_install)
        ## Try again for a locally installed package
        installed_packages = local_packages
  
***************
*** 610,616 ****
    endfor
  
    ## Write a new ~/.octave_packages
!   if (issuperuser ())
      if (length (remaining_packages) == 0)
        unlink (global_list);
      else
--- 738,744 ----
    endfor
  
    ## Write a new ~/.octave_packages
!   if (global_install)
      if (length (remaining_packages) == 0)
        unlink (global_list);
      else
***************
*** 665,672 ****
      src = fullfile (packdir, "src");
      ## configure
      if (exist (fullfile (src, "configure"), "file"))
!       [status, output] = system (strcat ("cd ", src, "; ./configure 
--prefix=",
!                                        desc.dir));
        if (verbose)
        printf("%s", output);
        endif
--- 793,804 ----
      src = fullfile (packdir, "src");
      ## configure
      if (exist (fullfile (src, "configure"), "file"))
!       [status, output] = shell (strcat ("cd ", src, "; ./configure 
--prefix=\"",
!                                        desc.dir, "\"",
!                                        " CC=", octave_config_info ("CC"),
!                                        " CXX=", octave_config_info ("CXX"),
!                                        " AR=", octave_config_info ("AR"),
!                                        " RANLIB=", octave_config_info 
("RANLIB")));
        if (verbose)
        printf("%s", output);
        endif
***************
*** 678,685 ****
  
      ## make
      if (exist (fullfile (src, "Makefile"), "file"))
!       [status, output] = system (strcat ("export INSTALLDIR=", desc.dir,
!                                        "; make -C ", src));
        if (verbose)
        printf("%s", output);
        endif
--- 810,817 ----
  
      ## make
      if (exist (fullfile (src, "Makefile"), "file"))
!       [status, output] = shell (strcat ("export INSTALLDIR=\"", desc.dir,
!                                        "\"; make -C ", src));
        if (verbose)
        printf("%s", output);
        endif
***************
*** 978,984 ****
  endfunction
  
  function out = issuperuser ()
!   out = strcmp (getenv("USER"), "root");
  endfunction
  
  ## This function makes sure the package contains the
--- 1110,1121 ----
  endfunction
  
  function out = issuperuser ()
!   if (ispc () && ! isunix ())
!     out = 1;
!   else
!     ## Need to be a bit presistent in probing superuser
!     out = (geteuid() == 0);
!   endif
  endfunction
  
  ## This function makes sure the package contains the
***************
*** 1275,1283 ****
    endif  
  
    ## Now check if the package is loaded
!   tmppath = path();
    for i = 1:length (installed_packages)
!     if (regexp (tmppath, installed_packages{i}.dir))
        installed_packages{i}.loaded = true;
      else
        installed_packages{i}.loaded = false;
--- 1412,1420 ----
    endif  
  
    ## Now check if the package is loaded
!   tmppath = strrep (path(), "\\", "/");
    for i = 1:length (installed_packages)
!     if (regexp (tmppath, strrep (installed_packages{i}.dir, "\\", "/")))
        installed_packages{i}.loaded = true;
      else
        installed_packages{i}.loaded = false;
***************
*** 1520,1522 ****
--- 1657,1685 ----
     octave_config_info("api_version")];
    arch = _arch;
  endfunction
+ 
+ function [status, output] = shell (cmd)
+ 
+   persistent have_sh;
+ 
+   cmd = strrep (cmd, "\\", "/");
+   disp (cmd);
+   if (ispc () && ! isunix ())
+     if (isempty(have_sh))
+       if (system("sh.exe -c \"exit\""))
+         have_sh = false;
+       else
+         have_sh = true;
+       endif
+     endif
+     if (have_sh)
+       [status, output] = system (["sh.exe -c \"", cmd, "\""]);
+     else
+       error ("Can not find the command shell")
+     endif
+   else
+     [status, output] = system (cmd);
+   endif
+   disp(output);
+ 
+ endfunction

2007-05-21  David Bateman  <address@hidden>

        * pkg/pkg.m: Add rebuild target, and -local, -global option to
        force installation location.
        (rebuild): New subfunction to rebuild package database from
        installed packages. Also allows changing of autoload status
        (install): Warn use if attempting to install from non existent
        file. 

2007-05-21  Michael Goffioul  <address@hidden>

        * miscellaneous/copyfile.m: check for cp.exe on Windows platforms
        and replace "\" characters with "/".
        * pkg/pkg.m: Use shell rather than system throughout. Replace "\" 
        characters with "/" throughout.
        (shell): New subfunction that wraps system and is careful with the
        shell on windows platforms.
        (configure_make): Explictly pass complier etc to configure
        process.
        (issuperuser): Use 'geteuid() == 0' instead of 'strcmp (getenv(
        "USER", "root"))' for root test.