Re: inadequate/inconsisten 'help pkg' message; problems with downloading

[Sergei Steshenko]

--- On Wed, 8/3/11, Sergei Steshenko <address@hidden> wrote:

> From: Sergei Steshenko <address@hidden>
> Subject: Re: inadequate/inconsisten 'help pkg' message; problems with 
> downloading
> To: address@hidden
> Date: Wednesday, August 3, 2011, 10:01 AM
> Trying to compile 'symbolic' package
> I've noticed two problems:
> 
> 1) it doesn't compile;
> 2) apparently wrong order of messages - please see in the
> attached file:
> 
> "
>      49 symbols.cc: In function
> ‘octave_value_list Fsubs(const octave_value_list&,
> int)’:
>      50 symbols.cc:376:17: error:
> ‘const class octave_value’ has no member named
> ‘is_list’
>      51 make: *** [symbols.o] Error 1
>      52 'make' returned the following
> error: make: Entering directory
> `/tmp/oct-XTeaMy/symbolic-1.0.9/src'
>      53 mkoctfile -v    -c
> symbols.cc
> "
> 
> - I think first there should be
> 
> mkoctfile -v    -c symbols.cc
> 
> and then the error message.
> 
> I've looked into source of 'pkg.m' and, if I understand
> correctly, it
> prints its messages stdout. While the compiler (g++/gcc)
> prints messages
> to stderr.
> 
> I've seen "countless" cases like this - because of mixture
> of stdout and
> stderr order of messages is violated. This is because by
> default stdout
> is buffered and stderr is unbuffered.
> 
> So, if my understandings are correct, it's a bug in 'pkg.m'
> and maybe
> other files - stdout output should be replaced with stderr
> one.
> 
> Regards,
>   Sergei.
> 

I have further investigated the issue of stdout/stderr messages and now I
am sure it's a core 'octave' bug, and I have a fix for it.

In octave-3.4.2/share/octave/3.4.2/m/pkg/pkg.m file one can find
'configure_make' function described on lines ##1293 .. 1425.

In that function this:

   1322       [status, output] = shell (cstrcat ("cd '", src, "'; ", scenv,
   1323                                          "./configure --prefix=\"",
   1324                                          desc.dir, "\"", flags));

piece of code calls 'configure' while this:

   1336       [status, output] = shell (cstrcat (scenv, "make -C '", src, "'"));

calls 'make'.

As one can see, both calls are performed through 'shell' function:

   2248 function [status, output] = shell (cmd)
   2249   persistent have_sh;
   2250
   2251   cmd = strrep (cmd, "\\", "/");
   2252   if (ispc () && ! isunix ())
   2253     if (isempty(have_sh))
   2254       if (system("sh.exe -c \"exit\""))
   2255         have_sh = false;
   2256       else
   2257         have_sh = true;
   2258       endif
   2259     endif
   2260     if (have_sh)
   2261       [status, output] = system (cstrcat ("sh.exe -c \"", cmd, "\""));
   2262     else
   2263       error ("Can not find the command shell");
   2264     endif
   2265   else
   2266     [status, output] = system (cmd);
   2267   endif
   2268 endfunction

- the same 'octave-3.4.2/share/octave/3.4.2/m/pkg/pkg.m' file.

What is important is that 'octave' core 'system' function is called, and
of its screen output _only_ stdout is saved in a variable, e.g.:

   2266     [status, output] = system (cmd);
.

So, if we get back to, say:

   1334     ## Make.
   1335     if (exist (fullfile (src, "Makefile"), "file"))
   1336       [status, output] = shell (cstrcat (scenv, "make -C '", src, "'"));
   1337       if (status != 0)
   1338         rm_rf (desc.dir);
   1339         error ("'make' returned the following error: %s", output);
   1340       elseif (verbose)
   1341         printf("%s", output);
   1342       endif
   1343     endif

and think how it works, the sequence is this:

1) 'make' is called in

   1336       [status, output] = shell (cstrcat (scenv, "make -C '", src, "'"));

2) while 'make' is running, its stdout _only_ is being accumulated in
'output' varianle, but its stderr is printed to screen immediately as it is
produced;

3) upon 'make' completion, if 'verbose' is set, stdout _only_ is printed
to screen by

   1341         printf("%s", output);

statement.

I.e. disorderly stdout/stderr output is there by construction - because of
the way 'system' works.

In order to resolve the issue I wrote my own equivalent of 'shell' and now
I am using it. The newly written function is called

__shell_with_grabbed_stdout_stderr__
.

I also wrote a couple of service functions for the above.

I continue working on 'pkg.m' and hope to release it when I'm done.

Meanwhile the newly introduced functions can be found below.

The functions below also make 'pkg.m' respect TMP environment variable
variable.

I run Linux, so Windows-related code is untested.

Regards,
  Sergei.

   2283 ##################################################################
   2284 #
   2285 # The following '__shell_with_grabbed_stdout_stderr__' is added by
   2286 # Sergei Steshenko - the purpose is to have orderly screen output
   2287 # produced by 'cmd'. The built-in 'system' grabs only stdout.
   2288 # In this file thi causes out of order output during 'configure'
   2289 # and 'make'.
   2290 #
   2291 # According to
   2292 # 
http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/redirection.mspx?mfr=true
   2293 # some_cmd >stdout.log 2>&1
   2294 # - I don't have Windows Octave to check that the whole thing
   2295 # works.
   2296 #
   2297 # The function is based on 'shell' function originally present in
   2298 # this file.
   2299 #
   2300 ##################################################################
   2301
   2302 function [tmp_dir_for_stdout_stderr, shell_stdout_stderr_file] = 
__create_tmp_dir_for_stdout_stderr___()
   2303   TMP = getenv("TMP");
   2304   if(numel(TMP) ~= 0)
   2305     if(exist(TMP, "dir") ~= 7)
   2306       error("TMP=%s directory does not exist - either create it or 
unset TMP environment variable", TMP);
   2307     endif
   2308
   2309     tmp_dir_for_stdout_stderr = tmpnam(TMP, "oct-");
   2310     last_dir_path_element = 
tmp_dir_for_stdout_stderr(rindex(tmp_dir_for_stdout_stderr, filesep) + 1:end); 
#fprintf(stderr, "__shell_with_grabbed_stdout_stderr__:
   2310 last_dir_path_element=%s\n", last_dir_path_element);
   2311   else
   2312     tmp_dir_for_stdout_stderr = tmpnam;
   2313   endif
   2314
   2315   shell_stdout_stderr_file = cstrcat(tmp_dir_for_stdout_stderr, 
"/shell_stdout_stderr.log"); # fprintf(stderr, 
"__shell_with_grabbed_stdout_stderr__: shell_stdout_
   2315 stderr_file=%s\n", shell_stdout_stderr_file);
   2316
   2317   if(numel(TMP) == 0)
   2318     [st, msg, msgid] = mkdir(tmp_dir_for_stdout_stderr);
   2319
   2320     if(st == 0)
   2321       error("could not create '%s' directory, system message: %s, 
msgid=%d", tmp_dir_for_stdout_stderr, msg, msgid);
   2322     endif
   2323   else
   2324     [st, msg, msgid] = mkdir(TMP, last_dir_path_element);
   2325
   2326     if(st == 0)
   2327       error("could not create '%s' directory, system message: %s, 
msgid=%d", tmp_dir_for_stdout_stderr, msg, msgid);
   2328     endif
   2329   endif
   2330
   2331   #fprintf(stderr, "__shell_with_grabbed_stdout_stderr__: 
shell_stdout_stderr_file=%s\n", shell_stdout_stderr_file);
   2332 endfunction
   2333
   2334 function [status, output] = 
__execute_command_in_shell__(shell_and_command)
   2335   [tmp_dir_for_stdout_stderr, shell_stdout_stderr_file] = 
__create_tmp_dir_for_stdout_stderr___();
   2336   [status, output] = system(cstrcat(shell_and_command, " >", 
shell_stdout_stderr_file, " 2>&1"));
   2337   [fid, msg] = fopen(shell_stdout_stderr_file, "r");
   2338   if(fid == -1)
   2339     error("cannot open '$s' file for reading, system message: %s", 
shell_stdout_stderr_file, msg);
   2340   endif
   2341
   2342   output = char(fread(fid));
   2343
   2344   fclose(fid);
   2345   [err, msg] = unlink(shell_stdout_stderr_file)
   2346   if(err)
   2347     warning("__shell_with_grabbed_stdout_stderr__: could not remove 
'%s' file\n", shell_stdout_stderr_file);
   2348   endif
   2349
   2350   [st, msg, msgid] = rmdir(tmp_dir_for_stdout_stderr);
   2351   if(st == 0)
   2352     warning("__shell_with_grabbed_stdout_stderr__: could not remove '%' 
directory\n", tmp_dir_for_stdout_stderr);
   2353   endif
   2354 endfunction
   2355
   2356 function [status, output] = __shell_with_grabbed_stdout_stderr__(cmd)
   2357   persistent have_sh;
   2358
   2359   cmd = strrep (cmd, "\\", "/");
   2360   # Sergei - the above is essentially wrong. On a UNIXish system '\'
   2361   # can appear as escape character, and in such a case it shouldn't
   2362   # be replaced with '/'. But I just copy-pasted the line from
   2363   # 'shell' function originally present in this file.
   2364
   2365   if(ispc() && !isunix())
   2366     if(isempty(have_sh))
   2367       if(system("sh.exe -c \"exit\""))
   2368         have_sh = false;
   2369       else
   2370         have_sh = true;
   2371       endif
   2372     endif
   2373
   2374     if(have_sh)
   2375       [status, output] = __execute_command_in_shell__(cstrcat("sh.exe 
-c \"", cmd, "\""));
   2376     else
   2377       error ("Can not find the command shell");
   2378     endif
   2379   else
   2380     [status, output] = __execute_command_in_shell__(cmd);
   2381   endif
   2382 endfunction