Binding to privileged ports with XNIO

A couple random ideas about how to bind privileged (<1024) port numbers using XNIO. It might be possible to utilize the UNIX domain socket API (slated for 1.2.0) and use the file descriptor-passing mechanism. A root-privileged server (written in C) could be started that binds to a UNIX domain socket on the filesystem, whose access permissions are limited. The XNIO-based application can use this domain socket to request bound ports and get the file descriptor back, which it would use to construct a server.

Another idea would be to turn it around, and have the XNIO process run a setuid root program (via ProcessBuilder) which binds the port and hands it back through a socket created by the Java program.

JBoss Marshalling Framework 1.0.0.Beta2 Released

I'm pleased to announce that the second beta of the new marshalling framework has arrived. The API is pretty much final at this point; now it's just a bug quest before the first CR is released in a couple weeks. For more information about the features of the framework, please see the original Beta1 announcement.

Download the JARs from the repository. The documentation is available here at the JBoss docs site. Please report any bugs in the JBoss Marshalling bug tracker.

Many thanks to Ron Sigal, author of the Java serializaiton-compatible module, and the one who flushed out some most all of the bugs from the Beta1 River protocol implementation that were fixed in this version.

Minor XNIO re-org for 1.2.0

For XNIO 1.2.0, I'm doing away with the separated API versus standalone JAR. My original notion was that the standalone JAR would contain the whole API, plus the Xnio bootstrapping class, plus at least one implementation (NIO right now, as that is currently the only complete implementation). But I think it will be simpler going forward to just add Xnio into the API itself, and ship an API JAR plus one JAR per implementation. So from now on (including any future snapshots), the JARs will follow this new structure instead.

XNIO 1.1.0.GA Released

XNIO 1.1.0.GA is available on the XNIO downloads page. Not much change since 1.1.0.CR1. Just a Javadoc update, and one interface rename: TcpClient has been renamed to TcpChannelSource (see XNIO-55 for more information). Also see the online Javadocs for usage information.

Have fun!

Feedback about Dave's Dumb Closures

I've gotten some feedback about my closure idea. Some of it bad, some of it... also bad. I thought I'd reply to all the comments in a new posting here rather than try to squeeze it all in to the crappy comment thing.

Rémi Forax said...

You can't get ride of exception transparency, at least if you want to keep checked exceptions.

Here is an interresting email from Neal Gafter:
http://mail.openjdk.java.net/pipermail/closures-dev/2008-July/000174.html

Do you have a way to write StringSwitch with your syntax/semantics ?

Rémi raises two separate points here, so I'll address them separately. Firstly, there's the question of "exception transparency". In response to this, please consider the following. First, in my proposal, the passed in code block may only be directly executed from within the receiving method. Therefore, the set of exceptions that may be thrown can be statically determined. The JVM can thus "safely" throw undeclared checked exceptions from the closure body as long as the calling method can handle it. The JVM would be able to verify this statically during bytecode validation.

Secondly, the question is whether one could use closures to write a StringSwitch as shown in Neal Gafter's email above. The short answer is: no. It is not possible to use closures in this way in Java. The reason is that Java has a linear stack; therefore a closure in Java may not outlive its lexical scope.

Now I know you're thinking, "Pfft, this is proof that BGGA closures are better". However this is where I point out that BGGA closures aren't closures, not in a JVM sense. They're just a syntax shortcut for anonymous inner classes. And the only way the so-called StringSwitch could possibly be implemented is basically as a syntactical wrapper around Map<String,Runnable>, with some magic to produce anonymous inner classes. This is not any different from what we have today; except possibly that today there is a rough correlation between amount of code written and efficiency of execution. With BGGA that all changes... it almost becomes easier to write bad code than good code.

Yardena said...

Hi,

In languages that have closures, they are often used for "delayed evaluation", wrapping a block of code in a closure and passing on to some other object makes the code execute when the other object needs it and not right now. So closure can be used as an event handler, command, etc.

Unless I missed something major in your proposal, it does not support such a usage, since you don't support references to closures (I think this is what Remi is implying too).

So to get the record straight - you propose something much less powerful than BGGA, and not equivalent to closures in other languages.

You can still have delayed execution; however the delayed execution must occur within the lifetime of the called method. This is, as I mentioned, a consequence of how lexical scoping works in Java. In languages that support "keeping" the closure past its declaration, the lexical scope is either not defined by the stack, or the stack is "forked" so that stack frames can be preserved even after they're out of scope. Java's stack is linear, and lexical scope is defined by the stack, so if you want closures, you're stuck with that. Of course BGGA does not have this limitation because BGGA isn't really closures. It's an abbreviated syntax over anonymous inner classes.

This all isn't to say that this problem is without solution. One could propose a JVM change to add support for forked stacks, for example. But I for one think that interfaces already solve this problem much more idiomatically (and honestly). Idiomatic because it's done that way today, and there's really nothing seriously wrong with the solution. Honest because you get what you pay for - you're saying you want to create an object, and that's what the JVM does - unlike BGGA in which you say you want to pass in a closure, and it goes ahead and creates an object anyway.

Ricky Clarkson said...

"Basically this translates to Java as two things: A lexically-scoped block of code, and a means to execute that code or pass it on to another method for delayed execution."

Your proposal seems not to fit that second part, the delayed execution.
Suppose we're using a closurific wrapper around SwingUtilities.invokeLater:

doLaterClosurely() {
. . some gui stuff;
}

How would your proposal work for this?

It would be handy if you suggested what Java 5 code your examples translate to.

Since I've answered the former question already, I'll jump right to the end: There is no Java 5 translation for my closure proposal. This is the whole point. Today the JVM does not have closures; after BGGA, the JVM still would not have closures. With my proposal, the JVM gets closures - real closures that really close over the actual lexical scope. This means support on a JVM level. My proposal simply assumes the minimum amount of change to make this so, which is limited to not adding support for forked stacks (which, by the way, would imply the need to deal with multiple threads accessing a single stack frame).

Again I need to emphasize that BGGA is moving away from the notion that what you write is what you get. Much like autoboxing, a mistake we continue to have to live with to this day, BGGA results in objects being automatically created on your behalf without your direct intention to do so. One end result of this is that brevity of code has a greatly reduced correlation to efficiency of bytecode or efficiency of execution. This is a mistake in my opinion. One reason that Java is considered to be so fast now is that the virtual machine maps very cleanly to a real machine, and the Java language maps cleanly to the virtual machine. So things that are easy to do in Java, by and large, can execute quickly on a real machine. Programmers who come from a C background, like myself, tend to find that things function and perform "as expected" on a real machine.

The advent of BGGA represents a departure in correlating language features with JVM capabilities, and thus the language no longer maps cleanly to a real machine. The natural side-effect of this is that you have to perform a lot of trickery (such as implicitly converting local variables to fields) to make it work. It is an undeniable fact that field accesses are slower than local variable accesses. It is an undeniable fact that a direct subroutine invocation (such as what you'd get with the JVM "jsr" instruction) is much faster than a virtual method lookup and invocation. Programs using BGGA closures are going to appear to be "slower" than programs that don't, because they will use more anonymous inner classes, and they will suffer from the implicit local variable-to-field conversion and virtual method invocation. I guarantee that this will be the case.

WarpedJavaGuy said...

Why are new language constructs so often perceived as mutilations? Is changing the semantics of existing keywords and constructs not a mutilation?

I actually have a fairly simple set of criteria here for Java. I have no problem with constructs that map to the virtual machine in simple, well-defined ways. Today, a local variable in Java means a local variable in the JVM. Even so-called "hacks" such as inner classes have a fairly clean JVM mapping when it comes down to it - they're still classes after all. Anonymous inner classes push things with their implicit copying of local variables into final fields. BGGA is over the line. With BGGA, you could measurably impede the performance of a method body (not to mention completely change 100% of the method bytecode) simply by putting a closure inside of it somewhere which accesses those local variables. This is what I call a "mutilation". You don't get what you pay for when you use BGGA closures.

All my proposal would require is either a way to define code blocks as an attribute of the calling method, with a way to correlate the lexical scope, or a way to jsr into other methods with parameters (see the "Nuts & Bolts" section of the original posting). This would make my closures a very simple mapping to JVM bytecode.

Thanks everyone for your questions - please feel free to send in more.

More uses for Dave's Dumb Closures

Truly pluggable locking strategies:

    public interface Locker {
        void hold() for myBlock();
    }

    public class SynchLocker {
        private final Object lock;
        public SynchLocker(Object lock) { this.lock = lock; }
        public void hold() for myBlock() {
            synchronized (lock) {
                myBlock();
            }
        }
    }

    public class LockLocker {
        private final Lock lock;
        public LockLocker(Lock lock) { this.lock = lock; }
        public void hold() for myBlock() {
            lock.lock();
            try {
                myBlock();
            } finally {
                lock.unlock();
            }
        }
    }

    ...later...
    Locker l = ...take your pick...;
    l.hold() {
        ...do stuff...
    }

Let's see you do that efficiently with anonymous inner classes or BGGA.

JBoss Marshalling Framework 1.0.0.Beta1 Released

As a part of the JBoss Remoting project (and hopefully several other projects as well), I've developed a separated marshalling (a.k.a. serialization) framework. This framework was inspired by the need for certain features unavailable with the standard Object*Stream classes:

• Pluggable class resolvers, making it easy to customize classloader policy, by implementing a small interface (rather than having to subclass the Object*Stream classes)


• Pluggable object replacement (also without subclassing)


• Pluggable predefined class tables, which can dramatically decrease stream size and serialization time for stream types which frequently use a common set of classes


• Pluggable predefined instance tables, which make it easy to handle remote references


• Pluggable externalizers which may be used to serialize classes which are not Serializable, or for which an alternate strategy is needed


• Customizable stream headers


• Each marshaller instance is highly configurable and tunable to maximize performance based on expected usage patterns


• A generalized API which can support many different protocol implementations, including protocols which do not necessarily provide all the above features


• Inexpensive instance creation, beneficial to applications where many short-lived streams are used


• Support for separate class and instance caches, if the protocol permits; useful for sending multiple messages or requests with a single stream, with separate object graphs but retaining the class cache

The default implementation, known as "River", is also available as a separate JAR.

This project does not have a home page, but the binaries have been uploaded to repositry.jboss.org and the Javadocs are available online at docs.jboss.org. The source code may be found in my sandbox repository for now, until it gets a permanent home.

Enjoy!

XNIO 1.1.0.CR1 Released

As the title states - 1.1.0.CR1 is available at the XNIO project download page. The main list of changes:

• Added API support for creating one-way and two-way pipes.


• Improved separation of the NIO implementation by moving packages and moving the factory classes to a XnioNio class.


• Add a java.util.concurrent.Future wrapper for IoFuture objects.


• Fix a bug with IoFuture.get() so that it throws a more specific CancellationException when an operation is cancelled.


• Replace SPI module with a more logical abstraction for provider implementations, including some abstract implementations.


• Make IoFuture.await() parameter order consistent with other standard timed wait methods.


• Servers changed to bind to 0.0.0.0 by default.


• Added some handy util classes for dealing with Closeable resources.


• Consolidated the IoFuture.Status value TIMED_OUT into WAITING, since the two are synonymous anyway, which will simplify IoFuture.Notifier implementation among other things.


• Added an abstract base class to allow implementation of one-line IoFuture.Notifier implementations.