“No, I didn’t test it. I skimmed the source.”

Well, skim again, and be sure not to skip this one:

http://svn.berlios.de/svnroot/repos/allegra/lib/http_reactor.py

which precisely provides chunked-encoding and decoding. As for handling the close signal in a persistent connection, you have a point.

So thanks for the review, even if you appear to do it with a less than friendly purpose ;-)

“”"All HTTP/1.1 applications MUST be able to receive and decode the “chunked” transfer-coding, and MUST ignore chunk-extension extensions they do not understand.”"”

Or:

“”"For compatibility with HTTP/1.0 applications, HTTP/1.1 requests containing a message-body MUST include a valid Content-Length header field unless the server is known to be HTTP/1.1 compliant.”"”

Or:

“”"Persistent connections provide a mechanism by which a client and a server can signal the close of a TCP connection. This signaling takes place using the Connection header field (section 14.10). Once a close has been signaled, the client MUST NOT send any more requests on that connection.”"”

Did you read the RFC before claiming you had written an HTTP/1.1 client?

“(…) that isn’t an HTTP/1.1 client.”

Did you test it?

If yes I would be delighted to hear about it ;-)

Btw, I suggest you read RFC 2616, because that isn’t an HTTP/1.1 client. I doubt it’s even HTTP/0.9 compliant.

Yes it is. Nevow Athena works, QED. Practicality beats purity, as you say.

That’s not to say I wouldn’t *like* an HTTP/1.1 server. twisted.web’s file upload support is distressingly bad. but web2 will get there eventually, and hopefully Nevow’s users won’t notice a major disruption when it does. Again: maybe you could be working on improving web2 rather than writing yet another, different, differently broken HTTP/1.1 implementation.

2. “You can dispatch a pipe…”

Yes, but dispatching is not communicating. Medusa was mis-factored: transport logic in the protocol, protocol logic in the application, everything requires subclassing rather than composition.

3. “You can wrap … dispatchers around … collectors”

You certainly _can_, but that is because it’s so easy to build adapters in Python whiich can pretend to be just about any kind of object, whether they’re sockets or virtual consoles. If you build a serious amount of infrastructure around this, then you are basically re-implementing Twisted anyway - except you still have to declare support for new transports with a combinatorial explosion of inheritance rather than method calls like ‘listenUDP(FakeStream(MyClientThatExpectsTCP()))’

4. PNS

A brief glance at the code reveals this in PNS_circle.handle_read:

# bogus UDP read event triggered by errors like
# ECONNRESET

The logic is repeated in PNS_UDP_peer.handle_read(), but without the comment explaining what’s going on. Further evidence that things are mis-factored within medusa (and therefore allegra): this violates DRY, application code is being polluted with repetitive platform quirks, and you’re not even consistently documenting the repetition.

Furthermore PNS doesn’t emulate reliable connections, e.g. provide a “socket” like API for dispatchers that expect a socket to wrap around. For an example of what I’m talking about, see http://divmod.org/trac/browser/trunk/Vertex/vertex/ptcp.py

You say you want code review of Allegra; well, this is the same review I did before writing Twisted, and my conclusion remains the same: Medusa is a great idea, but fundamentally flawed in its factoring of OS-handling and application-handling logic. This is a problem even on a single platform and a disaster once you have to reliably handle things like winsock. At the time I couldn’t articulate it as “transport/protocol separation” and my first few attempts were little better. The difference between then and now is that my alternative is already implemented now, and you could use it if you were interested in its numerous advantages :)

If you would like more useful feedback on application features like PNS, some docstrings would be nice. Having read all of the documentation and most of the code, I have no idea what any of it is supposed to do.

Thanks for that long comment. I’ll answer each point.

1. HTTP/1.0 is good enough

As long as you serve one resource per connection or can define the length of each of them. But not if you expect to server Chunked-Encoded resources with no predefined length, or support AJAX clients and any other kind of long-lived pipelining user agents.

Then you will need HTTP/1.1, exactly like I did.

2. Dispatching Pipes

Yes you can dispatch a pipe … on POSIX system at least. That’s what the select_trigger.py module uses on other OS than NT to trigger the select/poll call and thunk back threaded continuation into the asynchronous loop.

Actually, most file descriptor will work with select/poll and can be dispatched, including STDIO on Linux.

3. Serial ports, UDP tunnelling, Virtual Connections, etc …

On POSIX serial ports are just files, you can select/poll them like sockets. You may have to adapt the handle_read and handle_write to support exceptions specific serial fds, but it is possible and probably quite simple (I have not tried that).

And for economic and technical reasons, the only plateform for which you want to write network device driver applications is most probably Linux or anothe free UNIX variant. That’s what I would use to program network appliances, not NT.

Practicality beats purity.

I and probably most network peer developers need not the kind of features that make UDP tunnelling or connection virtualization *apparently* simple.

If your application is hosted on a private and managed network, then you can set up all kind of proxies and tunnels (like SOCKS of ssltunnel) without increasing the complexity of your code and without the development time and performance penalty of doing all that in Python instead of C.

Only applications hosted in a single Internet peer require to integrate connection virtualization or protocol tunneling.

But nothing in the asyncore design prevents that.

You can elaborate pretty complex async_core dispatchers and async_chat collectors wrapping each other like russian dolls. For instance that’s what Allegra’s http_reactor.py module does to collect chunked-encoded MIME body for HTTP/1.1 pipelines. This is a simplistic example of application protocol tunnelling, but it’s as real as another (and possibly of many more real-world applications than UDP tunnelling ;-).

Distributed network resource resolution

Finally, speaking of UPnP and P2P protocols, I suggest you have a deep look at the PNS metabase peer that is the application for which Allegra was forked from Medusa.

First because it’s a good example of asynchronous UDP and TCP multiplexers integrated in a complex statefull peer application and using threads to access synchronous API.

Second because its purpose is precisely to solve the problem of distributed network resource resolution on a public network:

http://laurentszyster.be/blog/pns/

to embrace and extend DNS with a Public Name System, safe for public use and free as in free beer *and* free speech.

Regards,

Not a huge number, but it has already outpaced certain legacy servers such as Netscape.

While your comment about the *general* problem was accurate, your analysis of *particular* problems were totally wrong, and foom picked them apart better than I could so I will let his comments stand.

twisted.web2 has been languishing because despite widespread interest in Twisted’s web stack, twisted.web has actually been “good enough” for most people using it, and nobody has yet been willing to invest substantial resources to motivate the current developers or provide their own.

Finally, it is not really accurate to say that dispatcher “supports pipes”. You cannot, for example, call recv() on a pipe, so dispatcher.recv() will simply explode. More importantly, protocol implementations *expect* to be able to call recv(), or other socket-specific methods, so a protocol designed to run over a socket will break if you try to run it over stdin.

Twisted can speak to a socket, to a serial port or to a virtualized connection just as easily. I really actually use this feature of Twisted all the time, it is not a hypothetical benefit. Most recently I’ve used it to embed an actually decent, tested-on-networks message-passing protocol over stdin for use as a job control protocol for subprocesess, but the major advantage of transport/protocol separation is the ability to do things like UDP connection tunnelling for peer-to-peer applications. Divmod’s peer to peer protocol, “Q2Q”, is possible largely because of this separation.

UPnP is not everywhere yet! To have an effective P2P mesh you have to deal with NAT in any way possible, and “any way possible” means multiplexing multiple protocol connections over a single TCP connection, issuing CONNECT commands to proxies, and yes, simulating reliable delivery over UDP. I find it bewildering that allegra is allegedly for “python on peers” and you have not figured this out yet, but I suppose you are in good company. BitTorrent, the most popular “P2P” network in the world, also doesn’t work unless the user learns about port numbers and router configuration.

I’ll start by noting that comments like “XXX”, calls to deprecated functions like “_setup_ssl”, the fact it lives in the contrib/ directory, and has only the most utterly trivial evidence of testing gives me very little confidence in that code.

Part of the difference appears to me to be that M2Crypto has a significantly different (and largely undocumented) API. e.g. ssl_read_nbio. Compare M2Crypto’s ssl_read_nbio code to Twisted’s _TLSMixin.doRead and you’ll see it has the same logic — the difference here is that Twisted is doing this logic itself, rather its SSL library taking care of it. Ditto for writes.

Also, I think that the large comment and associated tricky code in Twisted’s _sendCloseAlert would apply equally well to this code, but it doesn’t handle it.

Without digging any further, this already accounts for the difference in code size.

“I mean serial ports are supposed to drive devices like a console or a USB disk, aren’t they?”

So? There’s more to life than just networking. Event-driven protocols are a natural way to interact with, say, your GPS plugged in to your serial port, if your framework allows it.

Twisted is primarily focused on networking, but that’s not all it’s good for.

“Finally, note that asyncore dispatcher API does suite both named pipes, TCP sockets and to some extent regular file descriptors on POSIX systems (including STDIO’s).”

Twisted supports STDIO pipes fully, not just “to some extent”. I believe it even does so on Windows now as well.

However I still think that adding SSL/TLS support to asyncore dispatchers (be it for a TCP or UNIX pipe socket) is simpler than Twisted implementation:

http://svn.osafoundation.org/m2crypto/trunk/contrib/dispatcher.py

The whole issue of SSL/TLS decoupling is however quite moot and It’s a bit evil from my part to have waited for somebody else to invoke it (after all I *did* write once an asynchronous HTTPS/SOAP gateway)

First because there is little practical use for secure socket layers over anything else than TCP. Second because for a peer it may often be a wiser and safer choice to leave SSL/TLS implementation outside of the application and instead distribute that CPU intensive process on proxies (using ssltunnel for instance).

And third, when you need to implement secure socket layers in your peer, obviously practicality allways beats purity. Even in Twisted

The existence of a class _TLSMixin in:

http://twistedmatrix.com/trac/browser/trunk/twisted/internet/tcp.py

speaks volume about how SSL/TLS and TCP are actually coupled in its implementation of secure socket layers.

Now, I never wrote any protocol for serial ports, but I would expect those protocols to be of very different applications. I mean serial ports are supposed to drive devices like a console or a USB disk, aren’t they?

Finally, note that asyncore dispatcher API does suite both named pipes, TCP sockets and to some extent regular file descriptors on POSIX systems (including STDIO’s).

Regards,

How could this be overcome? Simply try to keep the synchronous nature of the code but without blocking. Before executing a blocking call, the sequential code calls
a method with the ressource(s) to wait upon and the condition (e.g. r/w) before
continuing.

I’ve implemented this with greenlets (a spin off of stackless, implemented as a CPython Extension module) where the waiting call suspends the current tasklet
and lets a scheduler (written in python) evaluate the ressource conditions (by doing the select/kevent/event and dispatching) and continue the tasklet when appropriate.

There is no need to save the state from local to instance variables
because the sequential code will be continued at the same place where it was
suspended by the “magic” waiting call for a ressource, just like generators in standard python (but these are restricted to yielding to the calling function which is not sufficient for this use case).

Unfortunately my code is not prime time ready but runs rather well for a multicast
upnp server with integrated http server which only uses the basic UDPServer and
SimpleHTTPServer without asyncore/asynchat or twisted and without blocking.

Twisted also supports SMTPS and its STARTTLS command. The TCP transport in Twisted can do the SSL handshake and transform itself into an SSL one if asked. But again, the SMTP code deals with the logic of the SMTP protocol, not how the bytes get delievered to it. (It can always ask the transport if it’s SSL and if so what the client certifcate and so on is if it cares about transport-specific features, but if it doesn’t care, well, it doesn’t have to care).

Being able to run HTTP over named pipes and write protocols that interact with serial port transports the exact same way you write any other byte-stream protocol is just a bonus ;)

It’s strange because I don’t understand the benefit of transport/protocol separation because SSL/TLS was specifically designed as a layer *added* on TCP socket. TCP/IP is the transport protocol, SSL is an *application* protocol.

The application is security.

That’s obvious if you consider SMTPS, because in this case the client *must* first perform an handshake *in clear* (using the commands STARTSSL or STARTTLS). Actually, SSL/TLS *may* be applied so that it appears as a transport layer as it does in HTTPS or FTPS, but then a peer TCP/TLS channel implementation may be enough, there is no need to decouple what *is* coupled.

Here is an example of an asynchronous HTTPS server developed with Medusa:

http://svn.osafoundation.org/m2crypto/trunk/demo/medusa054/https_server.py

which demonstrates that there is in this case no need to decouple HTTP and SSL/TLS in HTTPS. Hence, that there are probably not benefits to do so either.

If you don’t understand why Twisted has some abstractions, why not ask about them, rather than assume they are there just to make life harder? I think you’ll find they all exist for a reason, even if you can’t see it yet.

“You keep saying HTTP is a fundamental requirement of a network library.”

And indeed it is. The web protocol is the lingua-franca of the Internet and if they provide a network API most applications often use HTTP or piggy-back it because it is so pervasive. Wether it is a Web 2.0 service like del.icio.us or a P2P network like Gnutella’s.

It’s a good thing to have an asynchronous Python library for DNS, FTP, SMTP, NNTP, POP3, IMAP4, SIP, SOCKS, ICQ, MSN, OSCAR, SSH and telnet (or whatever …). But not if they are all of the same quality than the HTTP implemention found in Twisted.

“As for constructively criticising Allegra, it just doesn’t do anything interest enough to merit more than five minutes commenting on a blog post. Thoughtful code review takes a lot of effort that I would rather spend improving a useful project, rather than an ill-conceived also-ran.”

That’s harsh, but not much substantiated. Allegra may be ill-conceived (although I disagree with that) but it certainly is not an “also-ran”. And you sure won’t find out how usefull it may be until you actually read and/or test it.

To Bruce:

Twisted works and it has been reviewed and applied for a long time, enough to get stable. Yet, compared with asyncore, asynchat and Medusa, it looks un-necessarily complicated. I believed it was more practical and efficient to improve the standard Python asynchronous networking API rather than try to reinvent it.

The fact that Twisted is “undergoing fundamental architectural changes” may be an evidence that reinventing the wheel is seldom a good idea. Progress is all about marginal improvements, it comes much less often from a revolution.

Look at Python’s support for XML. Since the first Python binding to expat and Greg Stein’s

http://www.lyra.org/greg/python/qp_xml.py

we have seen a multiplication of DOMs and parsers developped. Yet, six years later what has been included in Python 2.5 standard library is … that unorthodox but damn practical element tree that Greg came up first.

I had that gut-feeling about Sam Rushing’s way too.