ANN: exn-source - exception backtraces with source code printing

[John Whitington]

Hi,

OCaml 4.02 has new facilities for installing a new backtrace handler,
and processing the back trace symbolically. As a little proof of
concept, I've built a package which you can add to any project to get
backtrace with source code printed and highlighted:

https://github.com/johnwhitington/exn-source

Here's the start of a backtrace:

Loading Image...

By default, it prints five lines either side, and looks the current
directory for source code. But as one can see from the screenshot, one
can add /usr/local/lib/ocaml to get stdlib source and so on...

It's somewhat difficult to test this kind of thing (we can't use
exceptions at all inside a backtrace handler, for example), so I'm
appealing for help. Attach exn-source to your code and let me know how
you get on...

Thanks,
--
John Whitington
Director, Coherent Graphics Ltd
http://www.coherentpdf.com/

[Gabriel Scherer]

That looks like a nice project ! Another trick I use personally is to run
the faulting program from inside Emacs's compile-mode (using "build && run"
instead of just "build" as a compilatoin command), and piggy-back on its
parsing of OCaml error locations to ask Emacs to drive me around the
relevant files.
I'm sure there are scenarios where your simultaneous view of all fragments
at once can be very helpful, and I'm also interested in the uses of the new
backtrace/callstack inspection API.

(we can't use exceptions at all inside a backtrace handler, for example)

I'm a bit surprised by this. Part of the point of reifying traces into
datatypes (raw_backtrace, backtrace_slot) was to make them persistent.
set_uncaught_exception_handler is passed a raw trace that should not be
mutated by raising new exceptions. What breaks if you use exceptions inside
a backtrace handler?

On Sun, Oct 19, 2014 at 8:03 PM, John Whitington <

Hi,
OCaml 4.02 has new facilities for installing a new backtrace handler, and
processing the back trace symbolically. As a little proof of concept, I've
built a package which you can add to any project to get backtrace with
https://github.com/johnwhitington/exn-source
http://www.coherentpdf.com/exnsource.png
By default, it prints five lines either side, and looks the current
directory for source code. But as one can see from the screenshot, one can
add /usr/local/lib/ocaml to get stdlib source and so on...
It's somewhat difficult to test this kind of thing (we can't use
exceptions at all inside a backtrace handler, for example), so I'm
appealing for help. Attach exn-source to your code and let me know how you
get on...
Thanks,
--
John Whitington
Director, Coherent Graphics Ltd
http://www.coherentpdf.com/
--
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

[John Whitington]

Hi Gabriel,

Post by Gabriel Scherer

(we can't use exceptions at all inside a backtrace handler, for example)

I'm a bit surprised by this. Part of the point of reifying traces into
datatypes (raw_backtrace, backtrace_slot) was to make them persistent.
set_uncaught_exception_handler is passed a raw trace that should not be
mutated by raising new exceptions. What breaks if you use exceptions
inside a backtrace handler?

To quote the documentation for set_uncaught_exception_handler: "If fn
raises an exception, it is ignored."

This is a bit ambiguous -- it might mean

a) The uncaught exception handler you register won't get called from
inside itself in the case of an uncaught exception. Nor will any such
exceptions be handled by the standard exception handler; or

b) Exceptions in the handler literally have no effect.

I've done a little test just now, and it seems to be (a), so perhaps
it's just in need of clarification in the documentation.

(It might be nice to have, for development, an option to have the
default exception handler remain active, so it reports any exceptions
escaping from the user-installed exception handler).

Thanks,
--
John Whitington
Director, Coherent Graphics Ltd
http://www.coherentpdf.com/

[Gabriel Scherer]

perhaps it's just in need of clarification in the documentation.

My understanding is that uncaught exception raised by the handler are
dropped/ignored.
Would you mind creating a mantis issue ( http://caml.inria.fr/mantis/ ) so
that we can discuss improving the documentation there?

Note that a way to side-step this issue entirely would be for your API to
provide something in the style of Printexc.print
handle : ('a -> 'b) -> 'a -> 'b
that handles any exception raised by the function application. Users would
be able to call this explicitly around their main processing loop (which is
the place where they often already handle exception-handling) instead of
delegating to a possibly-fragile mutable final handler.

My understanding is that exn-source currently operates by a side-effect at
link-time. I would rather have the choice between an explicit (unit ->
unit) registration function, and a side-effect-free handling function as
above.

On Sun, Oct 19, 2014 at 8:55 PM, John Whitington <

Hi Gabriel,

Post by John Whitington

(we can't use exceptions at all inside a backtrace handler, for

example)
I'm a bit surprised by this. Part of the point of reifying traces into
datatypes (raw_backtrace, backtrace_slot) was to make them persistent.
set_uncaught_exception_handler is passed a raw trace that should not be
mutated by raising new exceptions. What breaks if you use exceptions
inside a backtrace handler?

To quote the documentation for set_uncaught_exception_handler: "If fn
raises an exception, it is ignored."
This is a bit ambiguous -- it might mean
a) The uncaught exception handler you register won't get called from
inside itself in the case of an uncaught exception. Nor will any such
exceptions be handled by the standard exception handler; or
b) Exceptions in the handler literally have no effect.
I've done a little test just now, and it seems to be (a), so perhaps it's
just in need of clarification in the documentation.
(It might be nice to have, for development, an option to have the default
exception handler remain active, so it reports any exceptions escaping from
the user-installed exception handler).
Thanks,
--
John Whitington
Director, Coherent Graphics Ltd
http://www.coherentpdf.com/

[Gabriel Scherer]

Done. http://caml.inria.fr/mantis/view.php?id=6619

Thanks! Having an issue really helps tracking that things get done.

there is one advantage to the current approach: no source code needs to be

changed in the simplest case (just searching '.') - we just link exn-source
in.
[...]
So perhaps I should make it build exn-source.cm(x)a and exn-source-easy.cm(x)a,
and have two choices.

Indeed. exn-source-easy (or -autolink) could simply depend on the "clean
API" in the exn-source library, by being implemented as just
let () = Exn_source.register ()

Then simply making sure to pass the flag -linkall when building that second
cma means that explicitly requiring it (in the build system) will enable
the feature.

On Sun, Oct 19, 2014 at 9:38 PM, John Whitington <

Hi,

Post by Gabriel Scherer

perhaps it's just in need of clarification in the documentation.

My understanding is that uncaught exception raised by the handler are
dropped/ignored.
Would you mind creating a mantis issue ( http://caml.inria.fr/mantis/ )
so that we can discuss improving the documentation there?

Done. http://caml.inria.fr/mantis/view.php?id=6619
Note that a way to side-step this issue entirely would be for your API

Post by Gabriel Scherer
to provide something in the style of Printexc.print
handle : ('a -> 'b) -> 'a -> 'b
that handles any exception raised by the function application. Users
would be able to call this explicitly around their main processing loop
(which is the place where they often already handle exception-handling)
instead of delegating to a possibly-fragile mutable final handler.
My understanding is that exn-source currently operates by a side-effect
at link-time. I would rather have the choice between an explicit (unit
-> unit) registration function, and a side-effect-free handling function
as above.

Correct. Your suggestion would be much cleaner, but there is one advantage
to the current approach: no source code needs to be changed in the simplest
case (just searching '.') - we just link exn-source in.
People won't want to necessarily ship code with exn-source linked in, and
it's not nice to have to modify code between debug and release -- I like to
try to have those differences all in the build system.
So perhaps I should make it build exn-source.cm(x)a and exn-source-easy.cm(x)a,
and have two choices.
Thanks.
--
John Whitington
Director, Coherent Graphics Ltd
http://www.coherentpdf.com/

[Nicolas Boulay]

Very nive project !

May a suggest a different result presentation ? Most of the time, only name
of the function called are usefull, its more compact than 5 lines arround
the call site.

One other very important feature will be the parameter value of the
function call. It's value could be serialised on fixe size (<20 characters
by default). On a tricky bug, the next step is to print such value, so you
save time to print it with the function call.

Hi,
OCaml 4.02 has new facilities for installing a new backtrace handler, and
processing the back trace symbolically. As a little proof of concept, I've
built a package which you can add to any project to get backtrace with
https://github.com/johnwhitington/exn-source
http://www.coherentpdf.com/exnsource.png
By default, it prints five lines either side, and looks the current
directory for source code. But as one can see from the screenshot, one can
add /usr/local/lib/ocaml to get stdlib source and so on...
It's somewhat difficult to test this kind of thing (we can't use
exceptions at all inside a backtrace handler, for example), so I'm
appealing for help. Attach exn-source to your code and let me know how you
get on...
Thanks,
--
John Whitington
Director, Coherent Graphics Ltd
http://www.coherentpdf.com/
--
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

[Peter Zotov]

Post by John Whitington
Hi,
OCaml 4.02 has new facilities for installing a new backtrace handler,
and processing the back trace symbolically. As a little proof of
concept, I've built a package which you can add to any project to get
[snip]

Very nice project! However, it looks like the user needs to explicitly
specify the paths to the sources. I have some code to perform a similar
task[1]; the paths are actually contained inside the bytecode.
I think that for native-code you can fetch them in a similar way, though
it is a little harder to extract the debug info.

[1]:
https://github.com/whitequark/pry.ml/blob/master/src/pry_bytecode.ml,
see di_paths.

[John Whitington]

Hi,

Post by Peter Zotov

Post by John Whitington
Hi,
OCaml 4.02 has new facilities for installing a new backtrace handler,
and processing the back trace symbolically. As a little proof of
concept, I've built a package which you can add to any project to get
[snip]

Very nice project! However, it looks like the user needs to explicitly
specify the paths to the sources. I have some code to perform a similar

Yes, except for '.' which is hardcoded. So, in the simplest case, it works.

Post by Peter Zotov
task[1]; the paths are actually contained inside the bytecode.
I think that for native-code you can fetch them in a similar way, though
it is a little harder to extract the debug info.
[1]: https://github.com/whitequark/pry.ml/blob/master/src/pry_bytecode.ml,
see di_paths.

Noted. I'd like to add functionality like this. The first thing I'm
going to add is automatic finding of the stdlib, since this just
requires parsing the output of "ocamlc -config".

Unfortunately, I don't believe most OPAM packages install their source,
just the .mli files. I wonder how much extra space installing source
would take on average, and if people think it's worth it, for this or
other reasons.

Thanks,
--
John Whitington
Director, Coherent Graphics Ltd
http://www.coherentpdf.com/

[Peter Zotov]

Post by John Whitington
Noted. I'd like to add functionality like this. The first thing I'm
going to add is automatic finding of the stdlib, since this just
requires parsing the output of "ocamlc -config".

I would take this directly from OCaml configuration (module Config,
-package compiler-libs.common). No shelling out needed.

Post by John Whitington
Unfortunately, I don't believe most OPAM packages install their
source, just the .mli files. I wonder how much extra space installing
source would take on average, and if people think it's worth it, for
this or other reasons.

I think it's mainly due to lack of infrastructure that could install
sources (eg OASIS couldn't easily be coerced into doing that). You could
probably make opam do the grunt work for you; no need to have the
sources
installed in the same places as mli or binaries!

[Francois Berenger]

Post by Peter Zotov

Post by John Whitington
Noted. I'd like to add functionality like this. The first thing I'm
going to add is automatic finding of the stdlib, since this just
requires parsing the output of "ocamlc -config".

I would take this directly from OCaml configuration (module Config,
-package compiler-libs.common). No shelling out needed.

Post by John Whitington
Unfortunately, I don't believe most OPAM packages install their
source, just the .mli files. I wonder how much extra space installing
source would take on average, and if people think it's worth it, for
this or other reasons.

I think it's mainly due to lack of infrastructure that could install
sources (eg OASIS couldn't easily be coerced into doing that). You could
probably make opam do the grunt work for you; no need to have the sources
installed in the same places as mli or binaries!

Isn't opam -b or --keep-build-dir supposed to keep the sources?

[Török Edwin]

Post by Francois Berenger

Post by Peter Zotov

Post by John Whitington
Noted. I'd like to add functionality like this. The first thing I'm
going to add is automatic finding of the stdlib, since this just
requires parsing the output of "ocamlc -config".

I would take this directly from OCaml configuration (module Config,
-package compiler-libs.common). No shelling out needed.

Post by John Whitington
Unfortunately, I don't believe most OPAM packages install their
source, just the .mli files. I wonder how much extra space installing
source would take on average, and if people think it's worth it, for
this or other reasons.

Other reasons for wanting things from the build directory are 'cmt/cmti' files for merlin/ocamlspot/ocp-index (or their equivalent file that only stores documentation used by opam-doc).
Do these require the original source files to extract&format the documentation strings?

Post by Francois Berenger

Post by Peter Zotov
I think it's mainly due to lack of infrastructure that could install
sources (eg OASIS couldn't easily be coerced into doing that). You could
probably make opam do the grunt work for you; no need to have the sources
installed in the same places as mli or binaries!

Isn't opam -b or --keep-build-dir supposed to keep the sources?

It does, but it also requires more disk space by keeping all the binaries and intermediate files around.

Perhaps this tool could be extended to install the source files too, or something equivalent provided at the opam level:
https://bitbucket.org/camlspotter/spotinstall

Then you wouldn't need to modify the build system of every package to install its sources / its documentation /its annotation files.

Best regards,
--Edwin

[John Whitington]

Hi,

Post by John Whitington
Hi,
OCaml 4.02 has new facilities for installing a new backtrace handler,
and processing the back trace symbolically. As a little proof of
concept, I've built a package which you can add to any project to get
https://github.com/johnwhitington/exn-source
http://www.coherentpdf.com/exnsource.png
By default, it prints five lines either side, and looks the current
directory for source code.

Have cut a v0.1, now in OPAM "opam install exn-source".

a) Now finds the stdlib automatically;
b) Fixed to underline multiple-line errors;
c) Compiles with -linkall so activated merely by linking.

Thanks,
--
John Whitington
Director, Coherent Graphics Ltd
http://www.coherentpdf.com/