Library recompilation with OCamljava

[Kenneth Adam Miller]

So, I'm attempting a large library recompile with ocamljava. It's pretty
audacious, because some people in my workplace are rather unwilling to
learn ocaml, yet a very well respected and needed library is authored in
it. Everyone knows java, so we hope very much to recompile the library just
with ocamljava.

Two important things that make me nervous in pursuing this effort are:

1) once fully recompiled, will the library work in java as it did with
native/ocaml byte code?

I've already started modifying the popular OCamlMakefile project to add a
java-byte-code target, and I've found that a threaded example is admitted
by the compiler produces an uncaught exception when run on the JVM... That
example may fall out of what the ocamljava docs has been safely implemented
under ocamljava, I'm not sure...

2) I've noticed that the rather large library that I need to compile, as my
final target, and aside from the toy targets I'm testing ocamljava with,
actually consumes some other C libraries and functions in it's dependency
path...

This is difficult; the traditional OCamlMakefile builds traditional c stubs
to .o files, correctly compiled with ocamlc. But when put on the ocamljava
command line, but ocamljava doesn't know what to do about it. Would there
be any way that I can have support for this ocaml feature as well, or
facilitate some way to link in or enable ocaml code that calls into C being
compiled down to java?

3) What if the answer to 2) is no/no?

If I can't use ocamljava, which is the most desired and elegant way,
allowing beautiful language inter-operation, how can I *best* facilitate
calls to the ocaml library? Is there a fast way to generate callbacks to
ocaml in java or any other language? It's very highly preferable not to
have to delegate back through the JNI due to type safety and fragility.
Alternatively, I looked at the OCaml library Restful, and wondered to
myself if there could be any kind of fast definition between ocaml types
and an exchange language, like json or something. Ideally, I'd like to be
able to generate a url per function in a very very simple declarative
manner, so that I can take ocaml libraries, and make them operate as a
service, where ocaml library functions correspond to URLs.

[Kenneth Adam Miller]

So, after doing some more work, I think the answer to 2) is no/no. After
looking more into piqi, I now just need to find a way to do protocol
buffers based RPC to an ocaml service...

Anybody know how to do that?

On Thu, Oct 16, 2014 at 4:08 AM, Kenneth Adam Miller <

Post by Kenneth Adam Miller
So, I'm attempting a large library recompile with ocamljava. It's pretty
audacious, because some people in my workplace are rather unwilling to
learn ocaml, yet a very well respected and needed library is authored in
it. Everyone knows java, so we hope very much to recompile the library just
with ocamljava.
1) once fully recompiled, will the library work in java as it did with
native/ocaml byte code?
I've already started modifying the popular OCamlMakefile project to add a
java-byte-code target, and I've found that a threaded example is admitted
by the compiler produces an uncaught exception when run on the JVM... That
example may fall out of what the ocamljava docs has been safely implemented
under ocamljava, I'm not sure...
2) I've noticed that the rather large library that I need to compile, as
my final target, and aside from the toy targets I'm testing ocamljava with,
actually consumes some other C libraries and functions in it's dependency
path...
This is difficult; the traditional OCamlMakefile builds traditional c
stubs to .o files, correctly compiled with ocamlc. But when put on the
ocamljava command line, but ocamljava doesn't know what to do about it.
Would there be any way that I can have support for this ocaml feature as
well, or facilitate some way to link in or enable ocaml code that calls
into C being compiled down to java?
3) What if the answer to 2) is no/no?
If I can't use ocamljava, which is the most desired and elegant way,
allowing beautiful language inter-operation, how can I *best* facilitate
calls to the ocaml library? Is there a fast way to generate callbacks to
ocaml in java or any other language? It's very highly preferable not to
have to delegate back through the JNI due to type safety and fragility.
Alternatively, I looked at the OCaml library Restful, and wondered to
myself if there could be any kind of fast definition between ocaml types
and an exchange language, like json or something. Ideally, I'd like to be
able to generate a url per function in a very very simple declarative
manner, so that I can take ocaml libraries, and make them operate as a
service, where ocaml library functions correspond to URLs.

[f***@x9c.fr]

Hi Kenneth,

So, after doing some more work, I think the answer to 2) is no/no. After looking more into piqi, I now just need to find a way to do protocol buffers based RPC to an ocaml service...
Anybody know how to do that?
So, I'm attempting a large library recompile with ocamljava. It's pretty audacious, because some people in my workplace are rather unwilling to learn ocaml, yet a very well respected and needed library is authored in it. Everyone knows java, so we hope very much to recompile the library just with ocamljava.
1) once fully recompiled, will the library work in java as it did with native/ocaml byte code?
I've already started modifying the popular OCamlMakefile project to add a java-byte-code target, and I've found that a threaded example is admitted by the compiler produces an uncaught exception when run on the JVM... That example may fall out of what the ocamljava docs has been safely implemented under ocamljava, I'm not sure...

This sounds like a bug... It would be nice to report it at https://github.com/xclerc/ocamljava,
particularly if you have a (small) reproduction case. The "thread" library is indeed only lightly
tested, mainly because the "concurrent" library (which is specific to OCaml-Java) includes
another implementation of threads, that is much closer to Java threads.

2) I've noticed that the rather large library that I need to compile, as my final target, and aside from the toy targets I'm testing ocamljava with, actually consumes some other C libraries and functions in it's dependency path...
This is difficult; the traditional OCamlMakefile builds traditional c stubs to .o files, correctly compiled with ocamlc. But when put on the ocamljava command line, but ocamljava doesn't know what to do about it. Would there be any way that I can have support for this ocaml feature as well, or facilitate some way to link in or enable ocaml code that calls into C being compiled down to java?

OCaml-Java does not know how to handle C files, as additional primitives should be implemented
through Java files. As a consequence, when porting a project from ocamlc/ocamlopt to ocamljava,
you have to port C files to Java files.

3) What if the answer to 2) is no/no?
If I can't use ocamljava, which is the most desired and elegant way, allowing beautiful language inter-operation, how can I best facilitate calls to the ocaml library? Is there a fast way to generate callbacks to ocaml in java or any other language? It's very highly preferable not to have to delegate back through the JNI due to type safety and fragility. Alternatively, I looked at the OCaml library Restful, and wondered to myself if there could be any kind of fast definition between ocaml types and an exchange language, like json or something. Ideally, I'd like to be able to generate a url per function in a very very simple declarative manner, so that I can take ocaml libraries, and make them operate as a service, where ocaml library functions correspond to URLs.

I think a neat way to use a C library from an ocamljava-compiled program would be to have a
Java "backend" for Jeremy Yallop's ctypes (https://github.com/ocamllabs/ocaml-ctypes).
I never had the time to implement that, but toyed with this idea and think the best way to
implement it would be to go through JNA (https://github.com/twall/jna) rather than JNI.
JNA includes a "dlopen"-like mechanism, and automatically maps simple types from Java
to C. My knowledge of ctypes is quite limited, but I see no showstopper.


Regards,

Xavier

[Kenneth Adam Miller]

Ok, to replicate the bug, just go to
https://github.com/KennethAdamMiller/ocaml-makefile where I'm updating the
fork of ocamlmakefile in order to allow a target to compile for ocamljava.
Once I update (I've committed, but haven't pushed yet, will within a matter
of hours hours), you can clone or pull, and cd to the threads subdirectory,
which is an example project that consumes the ocamlmakefile. Then just do

make jbc #or make java-byte-code

ex1 and ex2 crashe with uncaught under/overflow exception and some other
overflow errors (not reliable); I think it's because the synchronization
that is expressed in the ocaml code isn't very well followed at java's
level; there seems to be some very great imbalance in thread scheduling
(possibly slow signal handling). In java, there's a long pause where the
output doesn't make it to the screen until the uncaught exception hits,
when it does, the numbers being passed have a huge difference. In contrast,
ocamlopt or ocamlc compiled code executes rather evenly, with each event
receiving scheduler attention pretty much equally. That's my jest.

Agreed! I'd much much rather construct some method of automatic (or even
just expedient manual intervention!) that offers ocaml -> C sublibrary :
java -> C sublibrary support! I've already started down the path of
ocamljava, and right now, even ocaml-RPC with protocol buffers looks like a
long haul as well! ocamljava is just a much better solution.

Precisely! ocaml-ctypes is exactly what's being used by the library that
I'm porting to call into C sub libraries. It would be really sweet if the
ocamljava compiler could detect the ocaml-ctypes and generate these
mappings automatically. This would eliminate a lot of error prone code,
since C code tends to interpret data raw at some point... How might I got
about writing this to boot? I'm moderately new to ocaml, lacking deep
expertise in it, but I'm an aggressive learner. Please explain the best
path forward, I want to create a robust and reusable solution.

Post by f***@x9c.fr
Hi Kenneth,
So, after doing some more work, I think the answer to 2) is no/no. After
looking more into piqi, I now just need to find a way to do protocol
buffers based RPC to an ocaml service...
Anybody know how to do that?
On Thu, Oct 16, 2014 at 4:08 AM, Kenneth Adam Miller <

Post by Kenneth Adam Miller
So, I'm attempting a large library recompile with ocamljava. It's pretty
audacious, because some people in my workplace are rather unwilling to
learn ocaml, yet a very well respected and needed library is authored in
it. Everyone knows java, so we hope very much to recompile the library just
with ocamljava.
1) once fully recompiled, will the library work in java as it did with
native/ocaml byte code?
I've already started modifying the popular OCamlMakefile project to add a
java-byte-code target, and I've found that a threaded example is admitted
by the compiler produces an uncaught exception when run on the JVM... That
example may fall out of what the ocamljava docs has been safely implemented
under ocamljava, I'm not sure...

This sounds like a bug... It would be nice to report it at
https://github.com/xclerc/ocamljava,
particularly if you have a (small) reproduction case. The "thread" library
is indeed only lightly
tested, mainly because the "concurrent" library (which is specific to OCaml-Java) includes
another implementation of threads, that is much closer to Java threads.
2) I've noticed that the rather large library that I need to compile, as

Post by Kenneth Adam Miller
my final target, and aside from the toy targets I'm testing ocamljava with,
actually consumes some other C libraries and functions in it's dependency
path...
This is difficult; the traditional OCamlMakefile builds traditional c
stubs to .o files, correctly compiled with ocamlc. But when put on the
ocamljava command line, but ocamljava doesn't know what to do about it.
Would there be any way that I can have support for this ocaml feature as
well, or facilitate some way to link in or enable ocaml code that calls
into C being compiled down to java?

OCaml-Java does not know how to handle C files, as additional primitives
should be implemented
through Java files. As a consequence, when porting a project from
ocamlc/ocamlopt to ocamljava,
you have to port C files to Java files.

Post by Kenneth Adam Miller
3) What if the answer to 2) is no/no?
If I can't use ocamljava, which is the most desired and elegant way,
allowing beautiful language inter-operation, how can I *best* facilitate
calls to the ocaml library? Is there a fast way to generate callbacks to
ocaml in java or any other language? It's very highly preferable not to
have to delegate back through the JNI due to type safety and fragility.
Alternatively, I looked at the OCaml library Restful, and wondered to
myself if there could be any kind of fast definition between ocaml types
and an exchange language, like json or something. Ideally, I'd like to be
able to generate a url per function in a very very simple declarative
manner, so that I can take ocaml libraries, and make them operate as a
service, where ocaml library functions correspond to URLs.

I think a neat way to use a C library from an ocamljava-compiled program would be to have a
Java "backend" for Jeremy Yallop's ctypes (
https://github.com/ocamllabs/ocaml-ctypes).
I never had the time to implement that, but toyed with this idea and think the best way to
implement it would be to go through JNA (https://github.com/twall/jna) rather than JNI.
JNA includes a "dlopen"-like mechanism, and automatically maps simple types from Java
to C. My knowledge of ctypes is quite limited, but I see no showstopper.
Regards,
Xavier

[Kenneth Adam Miller]

Ok, I've pushed to master-you can now pull and appreciate the new ocaml
target, jbc, which compiles for ocamljava. From there, recompiling each of
the examples provided (except calc, since it uses C, and that's under dev).

On Sat, Oct 18, 2014 at 6:04 AM, Kenneth Adam Miller <

Post by Kenneth Adam Miller
Ok, to replicate the bug, just go to
https://github.com/KennethAdamMiller/ocaml-makefile where I'm updating
the fork of ocamlmakefile in order to allow a target to compile for
ocamljava. Once I update (I've committed, but haven't pushed yet, will
within a matter of hours hours), you can clone or pull, and cd to the
threads subdirectory, which is an example project that consumes the
ocamlmakefile. Then just do
make jbc #or make java-byte-code
ex1 and ex2 crashe with uncaught under/overflow exception and some other
overflow errors (not reliable); I think it's because the synchronization
that is expressed in the ocaml code isn't very well followed at java's
level; there seems to be some very great imbalance in thread scheduling
(possibly slow signal handling). In java, there's a long pause where the
output doesn't make it to the screen until the uncaught exception hits,
when it does, the numbers being passed have a huge difference. In contrast,
ocamlopt or ocamlc compiled code executes rather evenly, with each event
receiving scheduler attention pretty much equally. That's my jest.
Agreed! I'd much much rather construct some method of automatic (or even
java -> C sublibrary support! I've already started down the path of
ocamljava, and right now, even ocaml-RPC with protocol buffers looks like a
long haul as well! ocamljava is just a much better solution.
Precisely! ocaml-ctypes is exactly what's being used by the library that
I'm porting to call into C sub libraries. It would be really sweet if the
ocamljava compiler could detect the ocaml-ctypes and generate these
mappings automatically. This would eliminate a lot of error prone code,
since C code tends to interpret data raw at some point... How might I got
about writing this to boot? I'm moderately new to ocaml, lacking deep
expertise in it, but I'm an aggressive learner. Please explain the best
path forward, I want to create a robust and reusable solution.

Post by f***@x9c.fr
Hi Kenneth,
So, after doing some more work, I think the answer to 2) is no/no. After
looking more into piqi, I now just need to find a way to do protocol
buffers based RPC to an ocaml service...
Anybody know how to do that?
On Thu, Oct 16, 2014 at 4:08 AM, Kenneth Adam Miller <

Post by Kenneth Adam Miller
So, I'm attempting a large library recompile with ocamljava. It's pretty
audacious, because some people in my workplace are rather unwilling to
learn ocaml, yet a very well respected and needed library is authored in
it. Everyone knows java, so we hope very much to recompile the library just
with ocamljava.
1) once fully recompiled, will the library work in java as it did with
native/ocaml byte code?
I've already started modifying the popular OCamlMakefile project to add
a java-byte-code target, and I've found that a threaded example is admitted
by the compiler produces an uncaught exception when run on the JVM... That
example may fall out of what the ocamljava docs has been safely implemented
under ocamljava, I'm not sure...

This sounds like a bug... It would be nice to report it at
https://github.com/xclerc/ocamljava,
particularly if you have a (small) reproduction case. The "thread"
library is indeed only lightly
tested, mainly because the "concurrent" library (which is specific to
OCaml-Java) includes
another implementation of threads, that is much closer to Java threads.
2) I've noticed that the rather large library that I need to compile, as

Post by Kenneth Adam Miller
my final target, and aside from the toy targets I'm testing ocamljava with,
actually consumes some other C libraries and functions in it's dependency
path...
This is difficult; the traditional OCamlMakefile builds traditional c
stubs to .o files, correctly compiled with ocamlc. But when put on the
ocamljava command line, but ocamljava doesn't know what to do about it.
Would there be any way that I can have support for this ocaml feature as
well, or facilitate some way to link in or enable ocaml code that calls
into C being compiled down to java?

OCaml-Java does not know how to handle C files, as additional primitives
should be implemented
through Java files. As a consequence, when porting a project from
ocamlc/ocamlopt to ocamljava,
you have to port C files to Java files.

Post by Kenneth Adam Miller
3) What if the answer to 2) is no/no?
If I can't use ocamljava, which is the most desired and elegant way,
allowing beautiful language inter-operation, how can I *best*
facilitate calls to the ocaml library? Is there a fast way to generate
callbacks to ocaml in java or any other language? It's very highly
preferable not to have to delegate back through the JNI due to type safety
and fragility. Alternatively, I looked at the OCaml library Restful, and
wondered to myself if there could be any kind of fast definition between
ocaml types and an exchange language, like json or something. Ideally, I'd
like to be able to generate a url per function in a very very simple
declarative manner, so that I can take ocaml libraries, and make them
operate as a service, where ocaml library functions correspond to URLs.

I think a neat way to use a C library from an ocamljava-compiled program
would be to have a
Java "backend" for Jeremy Yallop's ctypes (
https://github.com/ocamllabs/ocaml-ctypes).
I never had the time to implement that, but toyed with this idea and
think the best way to
implement it would be to go through JNA (https://github.com/twall/jna) rather than JNI.
JNA includes a "dlopen"-like mechanism, and automatically maps simple types from Java
to C. My knowledge of ctypes is quite limited, but I see no showstopper.
Regards,
Xavier

[Jeremy Yallop]

Post by Kenneth Adam Miller

Post by f***@x9c.fr
I think a neat way to use a C library from an ocamljava-compiled
program would be to have a Java "backend" for Jeremy Yallop's
ctypes (https://github.com/ocamllabs/ocaml-ctypes). I never had
the time to implement that, but toyed with this idea and think the
best way to implement it would be to go through JNA
(https://github.com/twall/jna) rather than JNI. JNA includes a
"dlopen"-like mechanism, and automatically maps simple types from
Java to C. My knowledge of ctypes is quite limited, but I see no
showstopper.

Precisely! ocaml-ctypes is exactly what's being used by the library
that I'm porting to call into C sub libraries. It would be really
sweet if the ocamljava compiler could detect the ocaml-ctypes and
generate these mappings automatically. This would eliminate a lot of
error prone code, since C code tends to interpret data raw at some
point... How might I got about writing this to boot? I'm moderately
new to ocaml, lacking deep expertise in it, but I'm an aggressive
learner. Please explain the best path forward, I want to create a
robust and reusable solution.

OCaml-Java support is on the wish list for ctypes

https://github.com/ocamllabs/ocaml-ctypes/issues/13

but we don't have the resources to implement it at present. Adding
support would mostly likely only involve changing ctypes itself, not
OCaml-Java, and is likely to involve writing OCaml-Java
implementations of the following components:

(1) memory access, i.e. functions for allocating blocks, for reading
and writing scalar values to arbitrary addresses, and for viewing C
objects as bigarrays:
https://github.com/ocamllabs/ocaml-ctypes/blob/master/src/ctypes/memory_stubs.ml
https://github.com/ocamllabs/ocaml-ctypes/blob/master/src/ctypes/bigarray_stubs.ml

(2) functions for printing primitive (scalar) values:
https://github.com/ocamllabs/ocaml-ctypes/blob/master/src/ctypes/value_printing_stubs.ml

(3) implementations of signed and unsigned integer types of various sizes:
https://github.com/ocamllabs/ocaml-ctypes/blob/master/src/ctypes/signed.ml
https://github.com/ocamllabs/ocaml-ctypes/blob/master/src/ctypes/unsigned.ml

(4) functions for converting between OCaml and C string representations
https://github.com/ocamllabs/ocaml-ctypes/blob/master/src/ctypes/std_view_stubs.ml

plus one of the following approaches for calling functions

(a) a "dynamic" approach, which resolves symbols and constructs call
frames at runtime, like the ctypes Foreign module. This involves two
components: a dynamic loading interface along the
following lines
https://github.com/ocamllabs/ocaml-ctypes/blob/master/src/ctypes-foreign-base/dl.mli
and primitives for dynamically constructing and making calls:
https://github.com/ocamllabs/ocaml-ctypes/blob/master/src/ctypes-foreign-base/ffi_stubs.ml

(b) a "static" approach, which generates code to be compiled by the
standard toolchain:
https://github.com/ocamllabs/ocaml-ctypes/blob/master/src/cstubs/cstubs.mli

Note that most of the links above are to internal Ctypes modules, not
to the interface, which I'd expect to remain largely unchanged.

As Xavier suggests, JNA may be a good starting point for some or all
of the above. If someone would like to look at adding OCaml-Java
support, please feel free to ask questions, either on the GitHub issue
tracker, or on the ctypes mailing list.

Jeremy.