Edrx's page on DaVinci (an IDE for Lua)
Ssshhh!!! The DaVinci project has not been
announced officially yet!
Please be discreet about this page, and
be aware that everything here may change...
Thanks!
Disclaimers:
(1): I haven't done all my homework on (learning) Scite yet - I know very little about it at this moment, and I haven't even followed the tutorials at the Lua wiki thoroughly - I had to do other things first.
(2): (this page is a rat's nest - get in touch)
(3): I'm temporarily working only on IupScintilla - it needs to be finished urgently.
(Last update: 2007oct08)
Quick index:
- 1. Overview
- 2. IUP/CD/IM
- 3. What we have now: DaVinci 0.0.1
- 4. The long way: select.lua and lcmd.lua, in and outside Scite
- 5. The short way: IUP using Scintilla as a text widget
- 6. Sending commands with SIGUSR2
- peek.lua
- On GDB, or: on little languages for discussing data structures
DaVinci will be an IDE for Lua based on SciTe/Scintilla and IUP. It will be composed of:
- a source code editor. This will initially be a slightly modified SciTe - SciTe is already scriptable in Lua, and can do syntax highlighting on Lua code - but in some months we also want to be able to control Scintilla directly from Lua, and to build other editors based on Scintilla, without SciTe.
- A debugger. This means running a Lua debugger (either ldb or remdebug, or both) from inside the editor, and having a separate window (like this screenshot) showing the current line being executed in the source code, breakpoints, values of some variables, etc. This separate window will be controlled by another graphical library - in early prototypes Tk, and later IUP.
- A "project manager" - maybe something like this screenshot. It should be able to produce a single executable file
Side-products:
Main news (details soon): I've been hired! An official announcement is due tonight (thursday, 2007jul19). We have some trivial running code - intro.lua (incomplete) - and sketches of lots of lots of what is going to be done.
(2007jul25): I've been postponing the official anouncement 8-\... I was afraid that people would think that the code here is too trivial and they wouldn't see where this is heading to, so I decided to add more examples to intro.lua before the release - ideally, I would put more examples of controlling SciTe and Scintilla from Lua, and examples of starting Tcl/Tk and IUP from Lua too - but then I discovered a problem with the event loop in Lua+libiuplua51 - and I need to explain that clearly before releasing the announcement...
1. Overview
Some people at TecGraf are trying to write an IDE for Lua based on IUP and Scintilla, and I'm trying to join them.
My e-scripts about DaVinci are here.
2007jul04: TCmd (screenshot here). This is a prototype code for something like the debug window of LuaCmd. The source explains the rationale and inner workings.
The DaVinci project: what we are trying to do
(2007jul08)
My first impression from reading the official specifications of the DaVinci project was that what we are trying to build is an IDE for Lua: that is, an editor, a debugger, and a "project manager" that keeps a list of source files, lets us edit any of them, and knows how to produce a single executable file from them all. All of this must be scriptable in Lua, of course...
The Department where Lua was created - TecGraf - also has other groups besides the Lua group, working on projects unrelated to Lua. Several of these projects are libraries that can be used directly from C; they do have Lua bindings, but they're independent from Lua.
The people who started the DaVinci project have created several of these graphical libraries - for example, IUP, CD, and IM - and a previous incarnation of DaVinci: LuaCmd. My impression as an outsider was that no one without direct contact with TecGraf was using those libraries; well, from this page we can see that the source code for some of these libraries has only become publically available a few years ago...
So, we will try to integrate DaVinci - the IDE - with these graphical libraries in several ways. The DaVinci program itself will be built on top of Scintilla (initially Scintilla and Scite), but parts of it - the "debug window", for example - will use IUP
2. IUP/CD/IM
One thing that made both TecGraf and a funding agency more interested in the DaVinci project was that it involves IUP.
The department where Lua was created - TecGraf - also has other groups besides the Lua group, working on projects unrelated to Lua. Some of these projects are libraries that can be used directly from C; they do have Lua bindings, but they are independent from Lua.
IUP, CD and IM are some of these libraries.
I used to have the impression that only people with close ties to TecGraf used IUP and CD and IM. Some years ago when I took a look at them their source code wasn't even publically available (confirm; see this), and I've tried to compile their current source packages recently and found that very hard. But a quick look at the LuaForge pages reveals that they get lots of downloads, so they can't be as "outsider-unfriendly" as I thought they were... anyway: one of my duties in the DaVinci project, as I am an outsider who thought that IUP and CD and IM were hard to use (because they are hard to compile, etc etc), is to fix these defects and make them "outsider-friendly" for more kinds of outsiders.
(To do: add links to what I already have - which is not much. This e-script and some things below it are what I use to unpack the precompiled ".so"s and load them into a running Lua interpreter; I didn't go very far in my attempts to recompile from the sources).The DaVinci project will use IUP in several ways.
- Some of our examples of running Lua programs from SciTe will be of programs that link with IUP and CD and IM.
- lcmd will use IUP for its GUI.
- (debugging Lua programs that link to IUP)
- At some point we want to be able to use Scintilla as a text widget for IUP; the versions of DaVinci that will use that will be much easier to port than the other ones.
3. What we have now: DaVinci 0.0.1
Status: version 0.0.1 is ready! It is just a proof of concept, it doesn't look like an IDE at all, and it will run only on *nix-likes (for reasons that we will see below) - but it has some nice running code! Take a look at the diagram below:
.________________ | | | emacs | | :: | | :: | | ____\/_______|__ ___________________ | | | | | | | shell ::::::::::::::::::::::::::::::::::> | | | | :: | | tcmd.tcl | |__|_____::_________| | (Tcl/Tk/Expect) | :: ^ | | :: : stdout | | ________\/_____:__ | | | : | | | | SciTe 1.74 : | | | | :: : | | | | ______\/____:__| | | | | | <--- /tmp/ee.tcmd.pid <--- | | | | Lua 5.1 | ---> /tmp/ee.tcmd.tcl | | | | | ------- kill -USR2 ------> | | |__|_______________| /tmp/ee.tcmd.tcl ---> |___________________| ^ ^ : : intro.lua : : lua50init.lua |
It corresponds to this screenshot:
4. The long way: select.lua and lcmd.lua, in and outside Scite
Here is what we expect to have running very soon: the Lua interpreter inside SciTe being able to the load IUP (and IM and CD) as libraries, and also Tcl/Tk in the same way; this Lua interpreter inside SciTe already has bindings to call functions from SciTe and Scintilla. On top of that it won't be hard to add a Lua debugger (ldb or remdebug), changing its current textual interfaces to a LuaCmd-like interface using a IUP or a Tk window. A diagram:
.__________________ | | | | Scintilla | | | /\ | | |_______::____| | ________________ | :: | | | | | SciTe 1.74 | | | Tk | | :: | | |_______| | ________\/_____| ___________|_____ /\ | | | | | | :: | | | Lua 5.1 ::::>| luatclbridge.so ::::> Tcl | | | | |_________________|__________| | | | _________________ __________ | | | | | | | | ::::>| libiuplua5.1.so ::::> IUP | | | | |_________________| CD | | | | | IM | |__|_______________| |________________| |
But how would the event loop of this work? We would have windows of three different types - Scintilla, Tk, IUP - and we'd need to handle events coming from any one of them... the event handlers are different, and the event handler in SciTe will have to deal with two other event handlers that are "on the other side of the Lua layer"...
One preliminary step is to implement this:
. ________________
| | |
| | Tk |
| |_______|
_______________ ___________|_____ /\ |
| | | | :: |
| ::::>| luatclbridge.so ::::> Tcl |
| Lua 5.1 | |_________________|__________|
| (lcmd 0.2) | _________________ __________
| | | | |
| ::::>| libiuplua5.1.so ::::> IUP |
| | |_________________| CD |
| | | IM |
|_______________| |________________|
|
Here the event handler will have to deal with events from two kinds of windows - Tk and IUP. Also, if we start a Lua interpreter in interactive mode and then we load luatclbridge.so and libiuplua5.1.so we don't want to lose the interactive prompt; so there's stdin/readline too, and, why not, let's add on top of that a signal handler, so that this Lua interpreter will also be able to react to, say, SIGUSR2s.
The Tcl people are always doing things like that (and there's the #tcl IRC channel at freenode!), so I will start by implementing this (lcmd 0.1):
. ________________
| | |
| | Tk |
| |_______|
_______________ ___________|_____ /\ |
| | | | :: |
| ::::>| luatclbridge.so ::::> Tcl |
| Lua 5.1 | |_________________|__________|
| (lcmd 0.1) |
| |
| |
| |
| |
|_______________|
|
I expect_ed_ to have this running by August 16 - but the priorities have changed (IupScintilla has to be finished soon).
5. The short way: IUP using Scintilla as a text widget
At some point - by november, I hope - we should also have this:
._____________...... | | : | Scintilla |<:::::::::::::::::::::::::::::::::. | | : :: |_____________| : :: : : :: : : :: : : :: : _______________: :: : | | :: : | Lua 5.1 | :: : | | :: : | | _________________ _____::___ : | | | | | : | ::::>| libiuplua5.1.so ::::> IUP | : | | |_________________| CD | : | | | IM | :..|_______________| |________________| |
here we will have a direct connection - in C, and independent from Lua, but with Lua bindings - between IUP and Scintilla; the idea is that IUP 3.0, that has not been released yet, should be able to use Scintilla as a text widget. And on top of that version of IUP we will implement a version of DaVinci that will not use SciTe.
This will be more portable and easier to install than the other implementations of DaVinci, but it is much less hacker-friendly than the other ideas... I will start with the other approaches, then produce prototypes in Lua for this one, then rewrite these prototypes in C.
6. Sending commands with SIGUSR2
The lower part of the diagram that describes DaVinci 0.0.1, above, shows a BIIIG arrow going rightwards; it is made of four sub-arrows, the uppermost of them going "in the wrong direction", from tcmd.tcl to Lua, while the other three go from Lua to tcmd.tcl. That big arrow - reproduced below - shows how tcmd.tcl receives commands from the outside:
. ___________________
| |
| |
_______________ | tcmd.tcl |
| | <--- /tmp/ee.tcmd.pid <--- | (Tcl/Tk/Expect) |
| Lua 5.1 | ---> /tmp/ee.tcmd.tcl | |
| | ------- kill -USR2 ------> | |
|_______________| /tmp/ee.tcmd.tcl ---> |___________________|
|
tcmd.tcl saves its pid to /tmp/ee.tcmd.pid, and then it waits
for SIGUSR2s; to send Tcl commands to it we write these commands
into /tmp/ee.tcmd.tcl, then we send a SIGUSR2 signal to
tcmd.tcl; then tcmd.tcl wakes up, reads the commands in /tmp/ee.tcmd.tcl, and executes them.
The code to listen for SIGUSR2s in tcmd.tcl is just this,
package require Expect; # "trap" and "pid" are Expect commands
proc pidfile {} { return /tmp/ee.tcmd.pid }
proc tclfile {} { return /tmp/ee.tcmd.tcl }
writefile [pidfile] "[pid]\n"
trap {source [tclfile]} USR2
|
and the code to send a command via SIGUSR2 to tcmd.tcl from Lua is just this (from intro.lua):
-- From: (find-angg "DAVINCI/intro.lua") -- See also: (find-angg "LUA/lua50init.lua" "getoutput") tcmd = function (tclcode) writefile("/tmp/ee.tcmd.tcl", tclcode) local output = getoutput("kill -USR2 $(cat /tmp/ee.tcmd.pid)") if output ~= "" then error(output) end end |
There are several reasons for not using communication by SIGUSR2s - it's not safe enough, that doesn't work on Windows outside
of Cygwin, "why don't you use sockets?", "why don't you use FIFOs?" -
but the code to implement communication by SIGUSR2s is trivial to
implement, so that's what I will use in the first prototypes.
Later all the libraries that need to talk to one another will be
linked into a single program, so they will call one another's
functions directly.
peek.lua
News (2007aug06): it took me 7 days to write this instead of the 4 that I expected (gaah! With so many urgent things 8-|), but a good part of peek.lua is ready -
# (find-es "davinci" "peek.lua:doc") # (find-angg "DAVINCI/peek.lua") # (find-angg "DAVINCI/peek.c") # (find-angg "DAVINCI/peek-luadecls-1.txt") # (find-angg "DAVINCI/peek-luadecls-2.txt") |
The current code is able to understand declarations of C types in a syntax that is very close to C (peek-luadecls-2.txt), but it still has some bugs (one is clearly marked in the code; a certain declaration of a union is parsed incorrectly - and I just learned that gcc handles alignment in stucts in a smart way, not in the most obvious way) and the part that will let us inspect C data structures from Lua is not ready yet, at all -
On GDB, or: on little languages for discussing data structures
(2007jul09)
I am not exactly a C programmer, but I somehow got involved with several programs that are written in C - Emacs, Lua, Scintilla, SciTe, IUP, Tcl/Tk - and whose data structures are quite complex.
Look at the figure at page 9 in this paper; if
you are familiar with upvalues and closures in Lua it gives you a very
good idea of how they are implemented, but if you try to use that
figure to understand the source code you will notice that something is
missing: the names of the data structures and of the fields. For
example: the boxes in the middle are "UpVal"s, the upward arrows
are "->mext"s, and the contents of those upvalues - if they
are strings - can be accessed with (rewrite and check) ...
(Write about the box representation of conses, how to extend it with new kinds of boxes for symbols and obarrays, and to add - and erase - addresses; discuss the Lua-ish representation; point to the two languages that can be used to access the data structures in a running program with GDB: just the commands, and commands-plus-some-gdb-output (this one would be good for e-mails); discuss the mapping between the structures in a C library and the Lua-visible structures than can be used from a Lua binding for that library).
(On scripting GDB: a message to a mailing list - here and here. What really matters is the screenshot in it - and it almost speaks by itself).
Scripting Scite and Scintilla using Lua
(Some messy notes are here, and above that block in the same page; there's a screenshot - of something trivial - here).
