Emacs and eev, or: How to Automate Almost EverythingEduardo Ochs
Quick Index:
Installing eev involves changing some rcfiles (using eev-rctool) (find-eev "eev-rctool") (find-angg "TH/eev-article.blogme" "making-progs-receive-cmds") but unistalling and undoing the changes to rcfiles is very easy. (find-eev "INSTALL") 1. Three kinds of interfacesInteractive programs in a Un*x system(1) can have basically three kinds of interfaces: they can be mouse-oriented, like most programs with graphical interfaces nowadays, in which commands are given by clicking with the mouse; they can be character-oriented, like most editors and mail readers, in which most commands are single keys or short sequences of keys; and they can be line-oriented, as, for example, shells are: in a shell commands are given by editing a full line and then typing “enter” to process that line. It is commonplace to classify computer users in a spectrum where the “users” are in one extreme and the “programmers” are in the other; the “users” tend to use only mouse-oriented and character-oriented programs, and the “programmers” only character-oriented and line-oriented programs. In this paper we will show a way to “automate” interactions with line-oriented programs, and, but not so well, to character-oriented programs; more precisely, it is a way to edit commands for these programs in a single central place --- Emacs --- and then send them to the programs; re-sending the same commands afterwards, with or without modifications, then becomes very easy. This way (“e-scripts”) can not be used to send commands to mouse-oriented programs --- at least not without introducing several new tricks. But “programmers” using Un*x systems usually see most mouse-oriented programs --- except for a few that are intrinsically mouse-oriented, like drawing programs --- as being just wrappers around line-oriented programs than perform the same tasks with different interfaces; and so, most mouse-oriented programs “do not matter”, and our method of automating interactions using e-scripts can be used to automate “almost everything”; hence the title of the paper. (1): Actually we are more interested in GNU systems than in “real” Unix systems; the reasons will become clear in the section nnn. By the way: the term “Unix” is Copyright (C) Bell Labs). 2. “Make each program do one thing well”One of the tenets of the Unix philosophy is that each program should do one thing, and do it well; this is a good design rule for Unix programs because the system makes it easy to invoke external programs to perform tasks, and to connect programs. Some of parts of a Unix system are more like “meta-programs” or
“sub-programs” than like self-contained programs that do some
clearly useful task by themselves. Shells, for example, are
meta-programs: their main function is to allow users to invoke “real
programs” and to connect these programs using pipes, redirections,
control structures (if, for, etc) and Unix “signals”. On the other
hand, libraries are sub-programs: for example, on GNU systems there's
a library called GNU readline that line-oriented programs can use to
get input; if a program, say, bc (a calculator) gets its input by
calling 3. Making programs receive commandsMany line-oriented programs allow “scripting”, which means
executing commands from a file. For example, in most shells we can say
“ So, it is possible to prepare commands for a shell (or for
scriptable line-oriented programs; for arbitrary line-oriented
programs see the section nnn) in several ways: by typing them at the
shell's interface --- and if the shell uses readline its interface can
be reasonably friendly --- or, alternatively, by using a text editor
to edit a file, say,
~/.bashrc , ~/.zshrc ,
...) we can reduce “source ~/ee.sh ” to just “ee ”: e , e , enter --- three keystrokes.
We just saw how a shell --- or, by the way, any line-oriented
program in which we can define an `ee' function like we did for the
shell --- can receive commands prepared in an external editor and
stored in a certain file; let's refer to that file, 4. Sending commandsGNU Emacs, “the extensible, self-documenting text-editor” ([S79]), does at least two things very well: one is to edit text, and so it can be used to edit temporary scripts, and thus to send commands to shells and to line-oriented programs with `ee' functions; and the other one is to run Lisp. Lisp is a powerful programming language, and (at least in principle!) any action or series of actions can be expressed as a program in Lisp; the first thing that we want to do is a way to mark a region of a text and “send it as commands to a shell”, by saving it in a temporary script file. We implement that in two ways:
ee ' (the name stands for something like `emacs-execute') just
saves the currently-marked region of text to ~/ee.sh ; `eev '
(for something like `emacs-execute-verbose') does the same but adding
to the beginning of the temporary script a command to put the shell in
“verbose mode”, where each command is displayed before being
executed, and also adding at the end an command to leave verbose mode.
We can now use ` 5. HyperlinksWhen we are using a system like *NIX, in a part of the time we are using programs with which we are perfectly familiar, and in the rest of the time we are using things that we don't understand completely and that make us have to access the documentation from time to time. In a GNU system the documentation is all on-line, and the steps needed to access any piece of documentation can be automated. We can use Emacs Lisp “one-liners” to create “hyperlinks” to files:
These expressions, when executed --- which is done by placing the
cursor after them and then typing
The convention is that these “extended hyperlink functions” have
names like ` Here are the definitions of `find-node' and `find-fline':
Now consider what happens when we send to a shell a sequence of commands like this one:
# ', that makes
the shell treat that line as a comment; but when we are editing that
in Emacs we can execute the `(find-node ...) ' with C-x C-e .
Hyperlinks can be mixed with shell code --- they just need to be
marked as comments.
Note: the actual definitions of ` (See also: Section \ref{e-scripts}) 6. Shorter HyperlinksThe hyperlinks in lines
A' , B' , C' in
Section 5, but are a bit shorter, and they hide details like Emacs's
path and the version of BusyBox; if we switch to newer versions of
Emacs and BusyBox we only need to change the definitions of `find-busyboxfile ' and `find-efile ' to update the hyperlinks.
Usually not many things change from one version of a package to
another, so most hyperlinks continue to work after the update.
Eev defines a function called `
The arguments for ` 7. Keys for following hyperlinks and for going back(Rewrite this; mention M-k, M-K, `to' and the (disabled) stubs to implement a `back' command) It is so common to have Lisp hyperlinks that extend from some position in a line --- usually after a comment sign --- to the end of the line that eev implements a special key for executing these hyperlinks: the effect of typing M-e (when eev is installed and “eev mode” is on) is roughly the same of first going to the end of the line and then typing C-x C-e; that is, M-e does the same as the key sequence C-e C-x C-e(1). (There are many other kinds of hyperlinks. Examples?) (1) The main difference between 8. Dangerous hyperlinksNote that these “hyperlinks” can do very dangerous things. If we start to execute blindly every Lisp expression we see just because it can do something interesting or take us to an interesting place then we can end up running something like:
The modern approach to safety in hyperlinks --- the one found in
web browsers, for example --- is that following a hyperlink can
execute only a few kinds of actions, all known to be safe; the
“target” of a hyperlink is something of the form Eev's approach is the opposite of that. I wrote the first functions of eev in my first weeks after installing GNU/Linux in my home machine and starting using GNU Emacs, in 1994; before that I was using mostly Forth (on MS-DOS), and I hadn't had a lot of exposure to *nix systems by then --- in particular, I had tried to understand *nix's notions of user IDs and file ownerships and permissions, and I felt that they were a thick layer of complexity that I wasn't being able to get through. Forth's attitude is more like ``the user knows what he's doing''; the system is kept very simple, so that understanding all the consequences of an action is not very hard. If the user wants to change a byte in a critical memory position and crash the machine he can do that, and partly because of that simplicity bringing the machine up again didn't use to take more than one minute (in the good old days, of course). Forth people developed good backup strategies to cope with the insecurities, and --- as strange as that might sound nowadays, where all machines are connected and multi-user and crackers abound --- using the system in the Forth way was productive and fun. *NIX systems are not like Forth, but when I started using them I
was accustomed to this idea of achieving simplicity through the lack
of safeguards, and eev reflects that. The only thing that keeps eev's
hyperlinks reasonably safe is transparency: the code that a
hyperlink executes is so visible that it is hard to mistake a
dangerous Lisp expression for a “real” hyperlink. Also, all the safe
hyperlink functions implemented by eev start with `
wget --help ”, puts the output of that in an Emacs
buffer and then jumps to the first occurrence of the string “recursive download ” there; other `find-xxxsh ' functions are
variations on that that execute some extra shell commands before
executing the first argument --- typically either switching to another
directory or loading an initialization file, like ~/.bashrc or
~/.zshrc . The `find-xxxsh0 ' functions are similar to their
`find-xxxsh ' counterparts, but instead of creating a buffer with
their output they just show it at Emacs's echo area and they use only
the first argument and ignore the others (the pos-spec).
9. Generating HyperlinksDo we need to remember the names of all hyperlinks functions, like
Eev implements several functions that create temporary buffers
containing hyperlinks, that can then be cut and pasted to other
buffers. For example, `
The first line of that buffer is a hyperlink to that
dynamically-generated page of hyperlinks. Its function --- `
The argument to `eek' is a string describing a sequence of keys in
a certain verbose format, and the effect of running, say, ((M-h is a prefix; ((Exceptions: M-h M-c, M-h M-2, M-h M-y. Show examples of how to edit hyperlinks with M-h M-2 and M-h M-y.)) ((Mention hyperlinks about a key sequence? ((Mention hyperlinks about a Debian package? 10. Returning from Hyperlinks((Mention M-k to kill the current buffer, and how Emacs asks for confirmation when it's a file and it's modified)) ((Mention M-K for burying the current buffer)) ((Mention what to do in the cases where a hyperlink points to the current buffer (section 16); there used to be an “ee-back” function bound to M-B, but to reactivate it I would have to add back some ugly code to `to'... (by the way, that included Rubikitch's contributions))) ((Web browsers have a way to “return” from hyperlinks: the “back” button... In eev we have many kinds of hyperlinks, including some that are unsafe and irreversible, but we have a few kinds of “back”s that work... 1) if the hyperlink opened a new file or buffer, then to kill the file or buffer, use M-k (an eev binding for kill-this-buffer); note that it asks for a confirmation when the buffer is associated to a file and it has been modified --- or we can use bury-buffer; M-K is an eev binding for bury-buffer. ((explain how emacs keeps a list of buffers?)) Note: if the buffer contains, say, a manpage, or an html page rendered by w3m, which take a significant time to generate, then M-K is better is than M-k. 2) if the hyperlink was a `to' then it jumped to another position in the same file... it is possible to keep a list of previous positions in a buffer and to create an `ee-back' function (suggestion: bind it to M-B) but I haver never been satisfied with the implementations that I did so we're only keeping a hook in `to' for a function that saves the current position before the jump)) ((dto recommended winner-undo)) 11. Local copies of files from the internetEmacs knows how to fetch files from the internet, but for most
purposes it is better to use local copies. Suppose that the
environment variable
emacs-paper.html inside ~/snarf/http/ . The two last lines are hyperlinks to the local copy;
`find-w3m ' opens it “as HTML”, using a web browser called w3m
that can be run either in standalone mode or inside Emacs; `find-w3m ' uses w3m's Emacs interface, and it accepts extra arguments,
which are treated as a pos-spec-list.
Instead of running the `
~/.psne.log ). It is more convenient to have a `psne ' that
changes the current directory of the shell than one that doesn't, and
for that it must be defined as a shell function.
Eev comes with an installer script, called
((See: http://lists.gnu.org/archive/html/eev/2005-06/msg00000.html)) 12. GlyphsEmacs allows redefining how characters are displayed, and one of
the modules of eev --- eev-glyphs --- uses that to make some
characters stand out. Character 15, for example, is displayed on the
screen by default as ' Eev changes the appearance of char 15 to make it be displayed as a
red star. Here is how: Emacs has some structures called “faces” that
store font and color information, and `
For this article, as red doesn't print well in black and white, we used this instead:
* ' (note that this is outside of the ascii range), using the default
face, i.e., the default font and color.
Eev also sets a few other glyphs with non-standard faces. The most
important of those are `
« ' and `» ' only
have character codes 171 and 187 in a few cases, mainly in the
`raw-text' encoding and in “unibyte” buffers; in most other
encodings they have other char codes, usually above 255, and when they
have these other codes Emacs considers that they are other characters
for which no special glyphs were set and shows them in the default
face. This visual distinction between the below-255 `« ' and ` ' and the other `« ' and `» 's is deliberate --- it helps
preventing some subtle bugs involving the anchor functions of section
\ref{anchors}.
(3). Determined by the “face” escape-glyph-face, introduced in GNU Emacs in late 2004. 13. Compose PairsTo insert a ` Eev-compose defines a few variables that hold tables of “compose
pairs”, which map pairs of characters that are easy to type into
other, weirder characters; for example, `eev-composes-otheriso' says
that the pair " The variable ` 14. Delimited regionsSometimes it happens that we need to run a certain (long) series of commands over and over again, maybe with some changes from one run to the next; then having to mark the block all the time becomes a hassle. One alternative to that is using a variaton on ` The original definition of eev-bounded was something like this:
ee-search-backwards ' searches for the first
occurrence of the string "\n#*\n " (newline, hash sign, control-O,
newline) before the cursor and returns the position after the "\n#*\n ", without moving the cursor; the call to ee-search-forward does something similar with a forward search. As
the arguments to `eev ' indicate the extremities of the region to
be saved into the temporary script, this saves the region between the
first "\n#*\n " backwards from the cursor to the first "\n#*\n " after the cursor.
The actual definition of `
Eev binds the key F3 to the function ` All these defaults values come from a single list, which is stored
in the variable `
Note that in Emacs Lisp (and in most other Lisps) each symbol has a
value as a variable that is independent from its “value as a
function”: actually a symbol is a structure containg a name, a
“value cell”, a “function cell” and a few other fields. Our
definition of ` Eev has an auxiliary function for defining these “bounded functions”; running
setq ' and the `defun '
above.
As for the meaning of the entries of the list ` Eev also implements other of these “bounded” functions. For
example, running `
ee-delimiter-percent ' holds the string "\n%*\n "; comments in LaTeX start with percent signs, not hash signs,
and it is convenient to use delimiters that are treated as comments.
((The block below ... tricky ... blah. How to typeset `
...for example eelatex, that saves the region (plus certain standard header and footer lines) to a “temporary LaTeX file” and saves into the temporary script file the commands to make `ee' run LaTeX on that and display the result. The block below is an example of (...) ...The block below shows a typical application of
((Comment about the size: the above code is “too small for being a script”, and the hyperlinks are important)) gdb (here-documents, gcc, ee-once) (alternative: here-documents, gcc, gdb, screenshot(s) for gdb) 15. Communication channelsThe way that we saw to send commands to a shell is in two steps:
first we use
The screenshot at Figure 3 shows this at work. The user has started
with the cursor at the second line from the top of the screen in the
Emacs window and then has typed F9 several times. Eev binds F9 to a
command that operates on the current line and then moves down to the
next line; if the current line starts with ` The first F9 executed ` The next two ` The next line had just ` In the following lines there is a small shell program that outputs
“ There are also ways to send whole blocks of lines at once through communication channels; see Section \ref{bigmodular}. 15.1. The Implementation of Communication ChannelsCommunication channels are implemented using an auxiliary script
called `eegchannel', which is written in Expect ([L90] and [L95]). If we start an xterm in the default way it starts a shell (say,
Eegchannel passes characters back and forth between the xterm and the shell without changing them in any way; it mostly tries to pretend that it is not there and that the xterm is communicating directly with the shell. However, when eegchannel receives a certain signal it sends to the shell a certain sequence of characters that were not sent by the xterm; it “fakes a sequence of keystrokes”. Let's see a concrete example. Suppose than Emacs was running with
process id (“pid”) 1000, and running
eegchannel A /bin/bash ”;
eegchannel saw the `A ', saved its pid (1002) to the file ~/.eev/eeg.A.pid , and watched for SIGUSR1 signals; every time
that it (the eegchannel) receives a SIGUSR1 it reads the contents
of ~/.eev/eeg.A.str and sends that as fake input to the shell
that it is controlling. So, running
echo $[1+2] ” (plus a newline)
“through the channel A”; what Emacs does when we type F9 on a line
that does not start with `* ' corresponds exactly to that.
16. AnchorsThe function `
What `to' does is simply to wrap its argument inside ` The function `
Actually ` 17. E-scriptsThe best short definition for eev that I've found involves some
cheating, as it is a circular definition: “eev is a library that adds
support for e-scripts to Emacs” --- and e-scripts are files that
contain chunks meant to be processed by eev's functions. Almost any
file can contain parts “meant for eev”: for example, a
When I started using GNU and Emacs the notion of an e-script was something quite precise to me: I was keeping notes on what I was learning and on all that I was trying to do, and I was keeping those notes in a format that was partly English (or Portuguese), partly executable things --- not all of them finished, or working --- after all, it was much more practical to write
Actually trying to define an e-script as being “a file containing executable parts, that are picked up and executed interactively” makes the concept of an e-script very loose. Note that we can execute the Lua parts in the code above by
running the Lua interpreter on it, we can execute the elisp
one-liner with A piece of code containing instructions in English on how to use it is also an e-script, in a sense; but to execute these instructions we need to invoke an external entity --- a human, usually ourselves --- to interpret them. This is much more flexible, but also much more error-prone and slow, than just pressing a simple sequence of keys like M-e, or F9, or F3, alt-tab, e, e, enter. 18. Splitting eev.elWhen I first submittted eev for inclusion in GNU Emacs, in 1999, the people at the FSF requested some changes. One of them was to split eev.el --- the code at that point was all in a single Emacs Lisp file, called eev.el --- into several separate source files according to functionality; at least the code for saving temporary scripts and the code for hyperlinks should be kept separate. It turned out that that was the wrong way of splitting eev. The frontier between what is a hyperlink and what is a block of commands is blurry:
The two ` So, what happens is that often a new kind of hyperlink will begin
its life as a series of shell commands (another example: using ` There's a much better way to split conceptually what eev does, though. Most functions in eev take a region of text (for example Emacs's own “selected region”, or the extent of Lisp expression coming before the cursor) and “execute” that in some way; the kinds of regions are
Actions (can be composed): * Saving a region or a string into a file * Sending a signal to a process * Executing as Lisp * Executing immediately in a shell * Start a debugger ((Emacs terminology: commands)) 19. Steps((Simple examples)) ((writing demos)) ((hyperlinks for which no short form is known)) ((producing animations and screenshots)) 20. Big Modular E-scriptsA shell can be run in two modes: either interactively, by expecting lines from the user and executing them as soon as they are received\footnote{except for multi-line commands.}, or by scripts: in the later case the shell already has access to the commands, and executes them in sequence as fast as possible, with no pause between one command and the next. When we are sending lines to a shell with F9 we are telling it not only what to execute but also when to execute it; this is somewhat similar to running a program step-by-step inside a debugger --- but note that most shells provide no single-stepping facilities. We will start with a toy example --- actually the example from Section \ref{anchors} with five new lines added at the end --- and then in the next section we will see a real-world example that uses these ideas.
((Somewhere between a script and direct user interaction)) ((No loops, no conditionals)) ((Several xterms)) 21. Internet Skills for Disconnected PeopleSuppose that we have a person P who has learned how to use a computer and now wants to learn how the internet works. That person $P$ knows a bit of programming and can use Emacs, and sure she can use e-mail clients and web browsers by clicking around with the mouse, but she has grown tired of just using those things as black boxes; now she wants to experiment with setting up HTTP and mail servers, to understand how data packets are driven around, how firewalls can block some connections, such things. The problem is that P has never had access to any machine besides her own, which is connected to the internet only through a modem; and also, she doesn't have any friends who are computer technicians or sysadmins, because from the little contact that she's had with these people she's got the impression that they live lifes that are almost as grey as the ones of factory workers, and she's afraid of them. To add up to all that, $P$ has some hippie job that makes her happy but poor, so she's not going to buy a second computer, and the books she can borrow, for example, Richard Stevens' series on TCP/IP programming, just don't cut. One of eev's intents isto make life easier for autodidacts. Can it be used to rescue people in positions like $P$'s(4)? It was thinking on that that I created a side-project to eev called “Internet Skills for Disconnected People”: it consists of e-scripts about running a second machine, called the “guest”, emulated inside the “host”, and making the two talk to each other via standard internet protocols, via emulated ethernet cards. Those e-scripts make heavy use of the concepts in the last section ((...))
(4). by the way, I created $P$ inspired on myself; my hippie job is being a mathematician. 22. Availability and ResourcesEev can be downloaded from the author's homepage, http://angg.twu.net/. That page also contains lots of examples, some animations showing some of eev's features at work, a mailing list, etc. Eev is in the middle of the process of becoming a standard part of GNU Emacs; I expect it to be integrated just after the release of GNU Emacs 22.1 in mid-2005. Eev's copyright has already been transferred to the FSF; it is distributed under the GPL license. 23. AcknowledgmentsI would like to thank David O'Toole, Diogo Leal and Leslie Watter for our countless hours of discussions about eev; many of the recent features of eev --- almost half of this article --- were conceived at our talks. ((Thank also the people at #emacs, for help with the code and for small revision tips)) 24. References[L90] - Libes, D. - Expect: Curing Those Uncontrollable Fits of Interaction. 1990. Available online from http://expect.nist.gov/. [L95] - Libes, D. - Exploring Expect. O'Reilly, 1995. [O99] - Ochs, E. - The Eev Manifesto (http://angg.twu.net/eev-manifesto.html). [S79] - Stallman, R. - EMACS: The Extensible, Customizable Display Editor. (http://www.gnu.org/software/emacs/emacs-paper.html) |