Re: Reading huge files

[Stefan Monnier]

> ;;; vlf.el --- View Large Files

I like that.

> ;; (defun vlf-extract-part-of-file (file from to)
> ;;   "Returns bytes in FILE from FROM to TO."
> ;;   (let ((size (vlf-file-size file)))
> ;;     (if (or (> from size)
> ;;             (> to size))
> ;;         (error "From or to is larger that the file size"))
> ;;     (with-temp-buffer
> ;;       (shell-command
> ;;        (format "head --bytes %d %s | tail --bytes %d"
> ;;           to file (+ (- to from) 1)) t)
> ;;       (buffer-substring (point-min) (point-max)))))

You'll indeed need something like that.  I'd recommend to use `dd' rather
than `split' or `head&tail'.  But note that the problem is bigger than you
think: the above will need to work with floating-point arguments (since
fixnums suffer from the 2^28 limit) and the %d format also suffers from the
2^28 limit (or 2^31 in Emacs-CVS).  That's another good reason to use `dd'
since you can use a 1048576 block size and thus divide your large floats by
1048576 before turning them into fixnums.

> (defvar vlf-mode-map (make-sparse-keymap)
>   "Keymap for `vlf-mode'.")

> (defun vlf-define-keymap ()
>   "Define keymap for `vlf-mode'."
>   (define-key vlf-mode-map [next] 'vlf-next)
>   (define-key vlf-mode-map [prior] 'vlf-prev)
>   (define-key vlf-mode-map "q" 'vlf-quit))

Please just use

  (defvar vlf-mode-map
    (let ((map (make-sparse-keymap)))
      (define-key map [next] 'vlf-next)
      (define-key map [prior] 'vlf-prev)
      (define-key map "q" 'vlf-quit)
      map)
    "Keymap for `vlf-mode'.")
 
Also, wrt features, it'd probably be good to "slide more smoothly":
e.g. always keep 2 or 3 blocks in the buffer at the same time so you can
comfortably look at the text at the boundaries between blocks.

Next step: use window-scroll-functions or jit-lock to detect when reaching
one of the ends so that we can automatically load in the next
consecutive block.


        Stefan