Call hack-local-variables from major modes rather than from file visiting

This prevents file/directory local variables from being lost when the major
mode is set or changed.

This fixes bug #15577 and bug #23407.

* lisp/files.el (normal-mode): Call `hack-local-variables' when the major mode
function hasn't already done so.
(hack-local-variables): Rename parameter `mode-only' to `handle-mode', make
its previous non-nil setting be t, and introduce the following action for a
non-nil non-t value: apply all settings apart from `mode'.

* lisp/subr.el (run-mode-hooks): call `hack-local-variables' for buffers
which are visiting files.

* doc/emacs/custom.texi (File Variables): Note that setting a major mode also
sets file variables.
(Directory Variables): Note that `mode', `eval', and `unibyte' can be set as
dir local variables, but `coding' can't.

* doc/lispref/modes.texi (Major Mode Conventions): Say that `run-mode-hooks'
also calls `hack-local-variables'.
(Auto Major Mode): Say that `find-file' no longer runs `hack-local-variables',
as from 25.2.  Remove vagueness from `normal-mode' and `set-auto-mode' by
saying that the mode IS SET, not merely "selected" or "chosen".
(Mode Hooks): Document change to `run-mode-hooks'.

* doc/lispref/variables.texi (File Local Variables): Document change to
`hack-local-variables'.

5 files changed, 81 insertions, 49 deletions

diff --git a/doc/emacs/custom.texi b/doc/emacs/custom.texi
index 5cb52e62ab..01637ae98a 100644
--- a/doc/emacs/custom.texi
+++ b/doc/emacs/custom.texi
@@ -1037,9 +1037,10 @@ explicitly.  For example, here's how to obtain the default value of
 @cindex file local variables
 
   A file can specify local variable values to use when editing the
-file with Emacs.  Visiting the file checks for local variable
-specifications; it automatically makes these variables local to the
-buffer, and sets them to the values specified in the file.
+file with Emacs.  Visiting the file or setting a major mode checks for
+local variable specifications; it automatically makes these variables
+local to the buffer, and sets them to the values specified in the
+file.
 
 @menu
 * Specifying File Variables:: Specifying file local variables.
@@ -1344,6 +1345,12 @@ be applied in the current directory, not in any subdirectories.
 Finally, it specifies a different @file{ChangeLog} file name for any
 file in the @file{src/imported} subdirectory.
 
+You can specify the variables @code{mode}, @code{eval}, and
+@code{unibyte} in your @file{.dir-locals.el}, and they have the same
+meanings as they would have in file local variables.  @code{coding}
+cannot be specified as a directory local variable.  @xref{File
+Variables}.
+
 @findex add-dir-local-variable
 @findex delete-dir-local-variable
 @findex copy-file-locals-to-dir-locals
diff --git a/doc/lispref/modes.texi b/doc/lispref/modes.texi
index ae79128f84..76e5174bd2 100644
--- a/doc/lispref/modes.texi
+++ b/doc/lispref/modes.texi
@@ -445,7 +445,8 @@ other packages would interfere with them.
 Each major mode should have a normal @dfn{mode hook} named
 @code{@var{modename}-mode-hook}.  The very last thing the major mode command
 should do is to call @code{run-mode-hooks}.  This runs the normal
-hook @code{change-major-mode-after-body-hook}, the mode hook,
+hook @code{change-major-mode-after-body-hook}, the mode hook, the
+function @code{hack-local-variables} (when the buffer is visiting a file),
 and then the normal hook @code{after-change-major-mode-hook}.
 @xref{Mode Hooks}.
 
@@ -525,11 +526,12 @@ the buffer based on information in the file name or in the file itself.
 It also processes local variables specified in the file text.
 
 @deffn Command normal-mode &optional find-file
-This function establishes the proper major mode and buffer-local variable
-bindings for the current buffer.  First it calls @code{set-auto-mode}
-(see below), then it runs @code{hack-local-variables} to parse, and
-bind or evaluate as appropriate, the file's local variables
-(@pxref{File Local Variables}).
+This function establishes the proper major mode and buffer-local
+variable bindings for the current buffer.  It calls
+@code{set-auto-mode} (see below).  As from Emacs 25.2, it no longer
+runs @code{hack-local-variables}, this now being done in
+@code{run-mode-hooks} at the initialization of major modes
+(@pxref{Mode Hooks}).
 
 If the @var{find-file} argument to @code{normal-mode} is non-@code{nil},
 @code{normal-mode} assumes that the @code{find-file} function is calling
@@ -543,9 +545,9 @@ If you run @code{normal-mode} interactively, the argument
 @var{find-file} is normally @code{nil}.  In this case,
 @code{normal-mode} unconditionally processes any file local variables.
 
-The function calls @code{set-auto-mode} to choose a major mode.  If this
-does not specify a mode, the buffer stays in the major mode determined
-by the default value of @code{major-mode} (see below).
+The function calls @code{set-auto-mode} to choose and set a major
+mode.  If this does not specify a mode, the buffer stays in the major
+mode determined by the default value of @code{major-mode} (see below).
 
 @cindex file mode specification error
 @code{normal-mode} uses @code{condition-case} around the call to the
@@ -555,16 +557,17 @@ mode specification error}, followed by the original error message.
 
 @defun set-auto-mode &optional keep-mode-if-same
 @cindex visited file mode
-  This function selects the major mode that is appropriate for the
-current buffer.  It bases its decision (in order of precedence) on the
-@w{@samp{-*-}} line, on any @samp{mode:} local variable near the end of
-a file, on the @w{@samp{#!}} line (using @code{interpreter-mode-alist}),
-on the text at the beginning of the buffer (using
-@code{magic-mode-alist}), and finally on the visited file name (using
-@code{auto-mode-alist}).  @xref{Choosing Modes, , How Major Modes are
-Chosen, emacs, The GNU Emacs Manual}.  If @code{enable-local-variables}
-is @code{nil}, @code{set-auto-mode} does not check the @w{@samp{-*-}}
-line, or near the end of the file, for any mode tag.
+  This function selects and sets the major mode that is appropriate
+for the current buffer.  It bases its decision (in order of
+precedence) on the @w{@samp{-*-}} line, on any @samp{mode:} local
+variable near the end of a file, on the @w{@samp{#!}} line (using
+@code{interpreter-mode-alist}), on the text at the beginning of the
+buffer (using @code{magic-mode-alist}), and finally on the visited
+file name (using @code{auto-mode-alist}).  @xref{Choosing Modes, , How
+Major Modes are Chosen, emacs, The GNU Emacs Manual}.  If
+@code{enable-local-variables} is @code{nil}, @code{set-auto-mode} does
+not check the @w{@samp{-*-}} line, or near the end of the file, for
+any mode tag.
 
 @vindex inhibit-local-variables-regexps
 There are some file types where it is not appropriate to scan the file
@@ -907,13 +910,14 @@ use the following functions to handle these conventions automatically.
 @defun run-mode-hooks &rest hookvars
 Major modes should run their mode hook using this function.  It is
 similar to @code{run-hooks} (@pxref{Hooks}), but it also runs
-@code{change-major-mode-after-body-hook} and
-@code{after-change-major-mode-hook}.
+@code{change-major-mode-after-body-hook}, @code{hack-local-variables}
+(when the buffer is visiting a file) (@pxref{File Local Variables}),
+and @code{after-change-major-mode-hook}.
 
 When this function is called during the execution of a
-@code{delay-mode-hooks} form, it does not run the hooks immediately.
-Instead, it arranges for the next call to @code{run-mode-hooks} to run
-them.
+@code{delay-mode-hooks} form, it does not run the hooks or
+@code{hack-local-variables} immediately.  Instead, it arranges for the
+next call to @code{run-mode-hooks} to run them.
 @end defun
 
 @defmac delay-mode-hooks body@dots{}
diff --git a/doc/lispref/variables.texi b/doc/lispref/variables.texi
index 6c53e9b6cc..dd3f18be4e 100644
--- a/doc/lispref/variables.texi
+++ b/doc/lispref/variables.texi
@@ -1613,7 +1613,7 @@ any form of file-local variable.  For examples of why you might want
 to use this, @pxref{Auto Major Mode}.
 @end defvar
 
-@defun hack-local-variables &optional mode-only
+@defun hack-local-variables &optional handle-mode
 This function parses, and binds or evaluates as appropriate, any local
 variables specified by the contents of the current buffer.  The variable
 @code{enable-local-variables} has its effect here.  However, this
@@ -1630,11 +1630,15 @@ is non-@code{nil}; it always calls the other hook.  This
 function ignores a @samp{mode} element if it specifies the same major
 mode as the buffer already has.
 
-If the optional argument @var{mode-only} is non-@code{nil}, then all
-this function does is return a symbol specifying the major mode,
-if the @w{@samp{-*-}} line or the local variables list specifies one,
-and @code{nil} otherwise.  It does not set the mode nor any other
-file-local variable.
+If the optional argument @var{handle-mode} is @code{t}, then all this
+function does is return a symbol specifying the major mode, if the
+@w{@samp{-*-}} line or the local variables list specifies one, and
+@code{nil} otherwise.  It does not set the mode or any other
+file-local variable.  If @var{handle-mode} has any value other than
+@code{nil} or @code{t}, any settings of @samp{mode} in the
+@w{@samp{-*-}} line or the local variables list are ignored, and the
+other settings are applied.  If @var{handle-mode} is @code{nil}, all
+the file local variables are set.
 @end defun
 
 @defvar file-local-variables-alist
diff --git a/lisp/files.el b/lisp/files.el
index 132ebced1c..d89b2f5258 100644
--- a/lisp/files.el
+++ b/lisp/files.el
@@ -2322,8 +2322,12 @@ in that case, this function acts as if `enable-local-variables' were t."
     ;; s-a-m and h-l-v may parse the same regions, looking for "mode:".
     (with-demoted-errors "File mode specification error: %s"
       (set-auto-mode))
-    (with-demoted-errors "File local-variables error: %s"
-      (hack-local-variables)))
+    ;; `delay-mode-hooks' being non-nil will have prevented the major
+    ;; mode's call to `run-mode-hooks' from calling
+    ;; `hack-local-variables'.  In that case, call it now.
+    (when delay-mode-hooks
+      (with-demoted-errors "File local-variables error: %s"
+        (hack-local-variables 'no-mode))))
   ;; Turn font lock off and on, to make sure it takes account of
   ;; whatever file local variables are relevant to it.
   (when (and font-lock-mode
@@ -3297,11 +3301,15 @@ DIR-NAME is the name of the associated directory.  Otherwise it is nil."
 ;; TODO?  Warn once per file rather than once per session?
 (defvar hack-local-variables--warned-lexical nil)
 
-(defun hack-local-variables (&optional mode-only)
+(defun hack-local-variables (&optional handle-mode)
   "Parse and put into effect this buffer's local variables spec.
 Uses `hack-local-variables-apply' to apply the variables.
 
-If MODE-ONLY is non-nil, all we do is check whether a \"mode:\"
+If HANDLE-MODE is nil, we apply all the specified local
+variables.  If HANDLE-MODE is neither nil nor t, we do the same,
+except that any settings of `mode' are ignored.
+
+If HANDLE-MODE is t, all we do is check whether a \"mode:\"
 is specified, and return the corresponding mode symbol, or nil.
 In this case, we try to ignore minor-modes, and only return a
 major-mode.
@@ -3319,7 +3327,7 @@ local variables, but directory-local variables may still be applied."
   (let ((enable-local-variables
 	 (and local-enable-local-variables enable-local-variables))
 	result)
-    (unless mode-only
+    (unless (eq handle-mode t)
       (setq file-local-variables-alist nil)
       (with-demoted-errors "Directory-local variables error: %s"
 	;; Note this is a no-op if enable-local-variables is nil.
@@ -3327,18 +3335,19 @@ local variables, but directory-local variables may still be applied."
     ;; This entire function is basically a no-op if enable-local-variables
     ;; is nil.  All it does is set file-local-variables-alist to nil.
     (when enable-local-variables
-      ;; This part used to ignore enable-local-variables when mode-only
-      ;; was non-nil.  That was inappropriate, eg consider the
+      ;; This part used to ignore enable-local-variables when handle-mode
+      ;; was t.  That was inappropriate, eg consider the
       ;; (artificial) example of:
       ;; (setq local-enable-local-variables nil)
       ;; Open a file foo.txt that contains "mode: sh".
       ;; It correctly opens in text-mode.
       ;; M-x set-visited-file name foo.c, and it incorrectly stays in text-mode.
       (unless (or (inhibit-local-variables-p)
-		  ;; If MODE-ONLY is non-nil, and the prop line specifies a
+		  ;; If HANDLE-MODE is t, and the prop line specifies a
 		  ;; mode, then we're done, and have no need to scan further.
-		  (and (setq result (hack-local-variables-prop-line mode-only))
-		       mode-only))
+		  (and (setq result (hack-local-variables-prop-line
+                                     (eq handle-mode t)))
+		       (eq handle-mode t)))
 	;; Look for "Local variables:" line in last page.
 	(save-excursion
 	  (goto-char (point-max))
@@ -3393,7 +3402,7 @@ local variables, but directory-local variables may still be applied."
 		  (goto-char (point-min))
 
 		  (while (not (or (eobp)
-                                  (and mode-only result)))
+                                  (and (eq handle-mode t) result)))
 		    ;; Find the variable name;
 		    (unless (looking-at hack-local-variable-regexp)
                       (error "Malformed local variable line: %S"
@@ -3410,7 +3419,7 @@ local variables, but directory-local variables may still be applied."
 		      (forward-char 1)
 		      (let ((read-circle nil))
 			(setq val (read (current-buffer))))
-		      (if mode-only
+		      (if (eq handle-mode t)
 			  (and (eq var 'mode)
 			       ;; Specifying minor-modes via mode: is
 			       ;; deprecated, but try to reject them anyway.
@@ -3432,6 +3441,7 @@ local variables, but directory-local variables may still be applied."
                                     ;; to use 'thisbuf's name in the
                                     ;; warning message.
                                     (or (buffer-file-name thisbuf) ""))))))
+                              ((and (eq var 'mode) handle-mode))
 			      (t
 			       (ignore-errors
 				 (push (cons (if (eq var 'eval)
@@ -3440,8 +3450,8 @@ local variables, but directory-local variables may still be applied."
 					     val) result))))))
 		    (forward-line 1))))))))
       ;; Now we've read all the local variables.
-      ;; If MODE-ONLY is non-nil, return whether the mode was specified.
-      (if mode-only result
+      ;; If HANDLE-MODE is t, return whether the mode was specified.
+      (if (eq handle-mode t) result
 	;; Otherwise, set the variables.
 	(hack-local-variables-filter result nil)
 	(hack-local-variables-apply)))))
diff --git a/lisp/subr.el b/lisp/subr.el
index afc86a77f8..f67f70f85c 100644
--- a/lisp/subr.el
+++ b/lisp/subr.el
@@ -1737,10 +1737,14 @@ if it is empty or a duplicate."
 
 (defun run-mode-hooks (&rest hooks)
   "Run mode hooks `delayed-mode-hooks' and HOOKS, or delay HOOKS.
-If the variable `delay-mode-hooks' is non-nil, does not run any hooks,
+Call `hack-local-variables' to set up file local and directory local
+variables.
+
+If the variable `delay-mode-hooks' is non-nil, does not do anything,
 just adds the HOOKS to the list `delayed-mode-hooks'.
 Otherwise, runs hooks in the sequence: `change-major-mode-after-body-hook',
-`delayed-mode-hooks' (in reverse order), HOOKS, and finally
+`delayed-mode-hooks' (in reverse order), HOOKS, then runs
+`hack-local-variables' and finally runs the hook
 `after-change-major-mode-hook'.  Major mode functions should use
 this instead of `run-hooks' when running their FOO-mode-hook."
   (if delay-mode-hooks
@@ -1751,6 +1755,9 @@ this instead of `run-hooks' when running their FOO-mode-hook."
     (setq hooks (nconc (nreverse delayed-mode-hooks) hooks))
     (setq delayed-mode-hooks nil)
     (apply 'run-hooks (cons 'change-major-mode-after-body-hook hooks))
+    (if (buffer-file-name)
+        (with-demoted-errors "File local-variables error: %s"
+          (hack-local-variables 'no-mode)))
     (run-hooks 'after-change-major-mode-hook)))
 
 (defmacro delay-mode-hooks (&rest body)