Allegra » Blog Archive » Back to the future, in 1996

Back to the future, in 1996

If you care, here is how Twisted is Broken compared to asyncore.

The when to read and/or write can no longer be implemented independantly from the how to do that or the what to do next.

There are other ways in which Twisted is Crippled compared to Medusa, but this one is the most significant for network protocol and application developers.

A Broken Couple

Twisted broke the Inversion of Control that enabled to decouple the event loop from both transport and protocol implementations.

By replacing the asyncore’s dispatcher methods:

readable ()
writable ()

with the four methods

stopReading ()
stopWriting ()
startReading ()
startWriting ()

of FileDescriptor in

twisted/internet/abstract

Twisted forces developers to call those four methods to include and exclude file descriptors from the I/O event loop. And do that in both the transport and protocol implementations.

When to read and/or write must be programmed where how to read and/or write and what to read and/or write are programmed.

Breaking everything

Twisted asynchronous I/O event interfaces are common dependencies of the transport and protocol implementations. Producers and consumers are therefore not able to stall channels without inheriting this dependency. It is therefore impossible to implement independantly a complex suite of encapsulable protocols, the essence of most Internet protocol.

Although transports are presented as decoupled from protocols, they are still both coupled to events by a common dependency. Transports and protocols are coupled to the event loop, there are enough parallel dependencies as evidence of that fact, for instance:

class FileDescriptor(...):
    ...
    producerPaused = 0
    ...

True, this Twisted framework can encapsulate many transports.

What is that good for network application developpers who will apply HTTP/1.1 and XML only, expecting the framework to chunk, unzip and decode MIME parts before beeing able to handle input asynchronously without having to worry about events?

What is that good for network protocol developpers that never use something else than TCP/IP as transport and yet expect a maximum of articulations to develop new protocols and applications with less code?

Without having to care about I/O events everywhere.

Twisted developpers cannot enjoy the practical benefits of coroutines as provided by asyncore for simple sockets, in 1996.

Ten years later, how is it possible that the when to read and/or write can no longer be implemented independantly from the how to do that or the what to do next?

Twisting asyncore broke it.

Warning

As Allways: verify local building code for suitability.

"There still remains the overhead of the readable() and writable() methods, I’m not sure what can be done about it… ideally we would like to have the object carefully manage the state of an attribute rather than requiring a method call. But I don’t think this can be retrofitted without breaking everything."
Sam Rushing: improvements (1999)

I’m not trying to hype Allegra or to spread FUD about Twisted. I’m not just bad even if I’m mean with code. I read both sources. I’m the original author of none. I’m describing how one works best and the other does not.

I forked Medusa for network peers because I saw no benefits in trying to fix Twisted. As I started to document Allegra’s core modules on this blog, a deeper understanding of Sam Rushing’s work sunk down my belgian chocolate brain.

I understand that there are applications actually developped with Twisted for which there is no way back. I can now explain more clearly how and why it is broken. But I cannot fix it.

Twisted applications must be fixed.

Now is better than never

Not right now. Back to the future, in 1996.

Don’t worry, all your memories of network protocols and applications up until 2006 will remain with you.

It’s a total recall.

This entry was posted on Friday, May 19th, 2006 at 6:00 pm and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. Responses are currently closed, but you can trackback from your own site.

21 Responses to “Back to the future, in 1996”

Allen Short Says:
May 19th, 2006 at 8:37 pm
Hmm.

So, even assuming you’re correct (which I’m not sure of yet), and the transport and protocol have to cooperate to “include and exclude file descriptors from the I/O event loop”….

Who cares?

Seriously, I have never wanted this, to my knowledge, in any program I have ever written, nor have I seen it needed in any program I have ever read. Do you have an example of how this matters to mortal network application developers?
Jean-Paul Says:
May 19th, 2006 at 8:55 pm
Hi Laurent. I think you have misunderstood the four methods of FileDescriptor which you mentioned. You say that protocol implementations are tied to these. However, if you investigate the Twisted codebase, you will not find any calls to them in protocol code:

$ grep –include ‘*.py’ -R -E ‘(start|stop)(Reading|Writing)’ twisted | grep -v ‘twisted/internet/’
twisted/conch/scripts/conch.py: def stopWriting(self):
twisted/conch/scripts/conch.py: def startWriting(self):
twisted/conch/scripts/cftp.py: def stopWriting(self):
twisted/conch/scripts/cftp.py: def startWriting(self):
twisted/conch/ssh/channel.py: self.startWriting()
twisted/conch/ssh/channel.py: self.stopWriting()
twisted/conch/ssh/channel.py: self.stopWriting()
twisted/conch/ssh/channel.py: def stopWriting(self):
twisted/conch/ssh/channel.py: def startWriting(self):
twisted/pair/tuntap.py: self.startReading()
twisted/pair/tuntap.py: self.stopReading()
twisted/web2/channel/fastcgi.py:# self.startReading()
$

We see a few definitions of methods with the same name (but they don’t actually serve the same purpose - the naming is coincidental) and a few calls to those methods. In fact, the only calls to the actual FileDescriptor methods we see are on implementations of new transports. There’s nothing wrong with this, since they are, after all, implementation details of transports in Twisted.

I’m glad you posted this, though. In looking for calls to these methods, I found two which were actually incorrect. I have since removed these in current Twisted trunk@HEAD.

I suppose the rest of your post stems from confusion over this point, since it doesn’t really make sense to me. Protocols don’t have to know anything about reading or writing, and in correctly implemented protocols (ie, those found in Twisted itself), *none* of them do. So your concerns about coupling and dependencies are unfounded. Twisted does not have the problems you describe.
Laurent Szyster Says:
May 19th, 2006 at 9:10 pm
To Allen:

Ok, let’s take a first example from FileDescriptor itself:

http://twistedmatrix.com/trac/browser/trunk/twisted/internet/abstract.py

go at line 53 to the connectionLost method. It is calling stopReading() and stopWriting(). On a close event, the channel cannot just flag its state as not connected, it must remove itself from the list of socket polled for I/O.

Another more practical example of the consequences, allways in FileDescriptor:

producerPaused = 0

at line 31 there is a class attribute obviously flagging that the producer is paused, would block, is stalled. Why? Can that state not be delegated to the producer itself? With Twisted it cannot.

Practicaly this yield monolithic implementations like:

http://www.mail-archive.com/twisted-web@twistedmatrix.com/msg00436.html

where stalling a producer is only possible by altering the state of the channel explicitely. Changing the state of the producer instance is not enough. There is a lot more funky code to write, yielding race conditions like

http://twistedmatrix.com/trac/ticket/811

The same is true for Twisted consumers.

Practically there is no way to develop on Twisted something like Medusa’s collectors, something to implement encapsulable protocols, independantly for output (producers) and input (collectors).

Practically asyncore enables to develop asynchat and support all those deeply encapsulated and pipelined text protocol.

With less code.
Laurent Szyster Says:
May 19th, 2006 at 9:27 pm
Hi Jean Paul,

Let me rephrase more correctly: Twisted developpers must call those four methods *if* they want to change the read/write state of the FileDescriptor in the I/O event loop.

The fact that they avoid and you consider a mistake to change that state in the protocol implementations is the proof that there is something broken.

You should be able to change that state in the protocol implementation, independantly from the transport implementation. With asyncore you can, because both are independant from the event loop.

An Allegra producer can stall its channel by changing its own state not having to call its channel to change its state, certainly not the list of file descriptors polled for I/O. That’s something the library takes care of, that you should understand, but which you may just not bother about.

With Twisted you cannot. Instead, protocols actually depend on transports for that critical state, they are tightly coupled.

Your findings just give more credit to the hypothesis that twisting asyncore broke it.
Jean-Paul Says:
May 19th, 2006 at 10:28 pm
Sorry, you’re still confused.

Let me adjust your rephrasing slightly so that it is correct:

Twisted developers *must not* call those four methods even if they want to change the read/write state of the FileDescriptor in the I/O event loop.

There is a very simple reason for this: those four methods are not part of any public API; they are implementation details which may be changed at any time. The real API for controlling whether a transport will deliver bytes to a protocol is the IProducer interface with its two methods pauseProducing and resumeProducing. There is no API for controlling whether bytes will be written to the underlying socket because such an API is not necessary with Twisted: it provides no functionality which is not available in some simpler way.

Now, I think we can forget about all that, because I don’t think your objection is actually to startReading/stopReading/startWriting/stopWriting. I think your real objection is that pauseProducing and resumeProducing are methods of a transport object instead of methods of the protocol object.

This is really an objection without any substance. If you want them to be methods on a protocol, then you can define them as such with a trivial mixin, which I release to you under the MIT license, feel free to make use of it any way you please:

class TransportStateMixin:
def pauseProducing(self): self.transport.pauseProducing()
def resumeProducing(self): self.transport.resumeProducing()

Your protocols can now use self.pauseProducing() and self.resumeProducing() and remain blissfully unaware of the protocol/transport separation that actually exists. Furthermore if you ever have a transport which is pausable and resumeable via some other API (not that I have ever come across such a thing, nor do I expect such a thing will ever exist), you can simply mix in a different class.

I suspect you still haven’t seen why this separation is superior to the mechanism asyncore employs. This, despite having quoted Sam Rushing’s explanation of the efficiency concerns associated with readable() and writable(). I’ll try to explain in simpler terms.

Each time it is going to do I/O, asyncore has to visit each dispatcher and invoke its readable() and writable() methods. It then builds up lists of dispatchers (and thus file descriptors) to poll for readability or writability. This is a relatively expensive task, particularly since it might involve calling arbitrary application-level logic which may be arbitrarily slow. And since polling for I/O events is something that happens very frequently, this cost builds up and results in lower performance than would be possible if these methods did not have to be called for each time around the loop.

The Twisted model differs in that it keeps these lists around all the time. It does not have to re-create them each time it checks for I/O events. I hope it is obvious why this is superior. Now, since the lists are not recreated every time, they have to be updated instead. This can be done in a variety of ways, but the most common way is to use pauseProducing() and resumeProducing(). Each of these is invoked by the protocol whenever it chooses, just as the readable() and writable() methods of Asyncore can change its return value whenever a protocol likes.

So the Twisted way is more efficient than the Asyncore way. But you aren’t primarily complaining about performance. You’re complaining about the coupling that this creates. I posit that the coupling is precisely the same as in Asyncore.

In Asyncore, the protocol must rely on the event loop to call its readable() method and then /not call handle_read/.

In Twisted, the protocol must rely on the event loop to /not call dataReceived/ if the transport’s pauseProducing has been called.

Do you see the similarity? Furthermore, have you considered what will happen if the protocol changes its mind between the event loop calls readable() and when the event loop delivers bytes to it with handle_read()? If you think this is impossible, consider the case of a proxy protocol where two different dispatchers rely on each other and one loses its connection in the same loop iteration as the other receives some data.

I hope this clears up why Twisted’s model is strictly an improvement on Asyncore’s and why the faults you pointed ou in Twisted are actually strengths.
Laurent Szyster Says:
May 20th, 2006 at 12:03 am
To Jean Paul:

Thanks for the summary of the argument:

“In Asyncore, the protocol must rely on the event loop to call its readable() method and then /not call handle_read/.”

True. And that event loop is not about to change.

What you can change is the implementation of readable() to suite your flavor of handle_read method for transport.

The asynchat flavor decouples further, and implements readable() so that you can also delegate readability to the protocol’s collector and writeability to the producer. And asynchat it decouple twice, allowing for independant implementation of input and output protocols.

And all that without ever having to depend on calling handle_read or handle_write.

They are called.

Once.

The alternative is:

“In Twisted, the protocol must rely on the event loop to /not call dataReceived/ if the transport’s pauseProducing has been called.”

I see the inversion, not the similarity.

The pauseProducing *has* been called so that the

FileDescriptor.dataReceived

method may not be called, so that the event could not be fired.

And pauseProducing must now be *called* by everything that would like to stall producing, that is blocking. Wether transport or protocol. And in a network peer there are plenty of I/O stalling. And that’s just one aspect of protocol development. What about protocol encapsulation and I/O decoupling for pipelined applications?

The old asyncore calls readable () and writable () once in the sources and once per dispatcher at runtime. There is no need to bother about read/write events anywhere else than in the few implementations of those interfaces, like asynchat.

The Medusa version of an HTTP/1.1 server allready came with an implementation of readable () that lets you push a producer instance that implements a protocol, and let it implement the readability condition the way it wants (if it wants).

The price to pay for all that power is to compute the state of a dispatcher in each run of the loop is.

Practically, it the time spent is probably smaller than calling Twisted four interfaces through a few layers of function calls each time a transport or protocol must change its state in the list.

And it possibly scales as well if not better anyway, because under high load there are as usually many events as there are dispatchers.

On the other hand the price Twisted source pays to achieve an ellusive performance, is a complicated code base which is impractical to develop forward. Not toward the web as it was implemented.

Five years no HTTP/1.1 peer. Why?

Because you cannot implement boxable collectors and producers for Twisted peers. Not without dependencies everywhere on pauseProducing. And, somehow, through a few useless articulations, that depend on the four twisted interfaces that thighly couple to the event loop *everything* that is developped on the FileDescriptor.

You have it under your own eyes and you can better than me reduce the fundamental differences.

Can you see the light?
Jean-Paul Says:
May 20th, 2006 at 12:26 am
> True. And that event loop is not about to change.

Neither is the transport interface in Twisted. :)

> I see the inversion, not the similarity.

It isn’t an inversion. In order to make your readable() method start returning True instead of False or False instead of True, you will have to change some state. Perhaps you set an attribute, or perhaps you append to a list, or any of a number of other things. The only difference with Twisted is you call a method instead of doing any of these other ad hoc things.

> And pauseProducing must now be *called* by everything that would like to stall producing, that is blocking. Wether transport or protocol. And in a network peer there are plenty of I/O stalling.

Yep. One uniform interface for one single task. Just like readable() is one uniform interface in Asyncore.

> What about protocol encapsulation and I/O decoupling for pipelined applications?

What about it? There are quite a few Twisted implementations of pipelined protocols. Nothing about Twisted’s model precludes this feature. It’s quite easy to implement.

> The Medusa version of an HTTP/1.1 server allready came with an implementation of readable () that lets you push a producer instance that implements a protocol, and let it implement the readability condition the way it wants (if it wants).

Twisted’s implementation of HTTP 1.0 supports arbitrary producers. So does Twisted’s implementation of FTP, POP3, IMAP4 and various other protocols. Furthermore, Twisted’s process support lets a producer stream results to a child process or to standard output.

> Practically, it the time spent is probably smaller than calling Twisted four interfaces through a few layers of function calls each time a transport or protocol must change its state in the list.

I recommend you perform some benchmarks before guessing about performance. I talked about efficiency concerns because I have done the benchmarks and I know the results. Twisted scales up better than Asyncore. This isn’t a theoretical problem.

> On the other hand the price Twisted source pays to achieve an ellusive performance, is a complicated code base which is impractical to develop forward.

I’m not sure if you meant “elusive” or “illusive” here ;) Either way, the price paid is negligable or entirely non-existent. The API is easily as convenient as Asyncore’s, and the performance benefits are real.

> Five years no HTTP/1.1 peer. Why?

Because it was never a priority for anyone. I recognize that HTTP/1.1 is important to you, but that doesn’t mean it is important to Twisted’s developers. We had an HTTP/1.0-ish implementation years ago, and it’s been good enough that HTTP/1.1 hasn’t been worth bothering with. The difficulties of implementing HTTP/1.1 aren’t just constrained to the quality of streaming support or the quality of encapsulation or the clarity of separation of concerns. It’s a fairly complicated protocol.

Now, recently someone has taken an interest in it. twisted.web2 *does* have an HTTP/1.1 implementation. It’s certainly not a finished, polished product but should demonstrate that once someone decided it was worth having, creating it was just a matter of investing a bit of developer time.

> Because you cannot implement boxable collectors and producers for Twisted peers. Not without dependencies everywhere on pauseProducing. And, somehow, through a few useless articulations, that depend on the four twisted interfaces that thighly couple to the event loop *everything* that is developped on the FileDescriptor.

Just not true. Take a look at the web2 code. None of it is coupled to FileDescriptor at all.

> You have it under your own eyes and you can better than me reduce the fundamental differences. Can you see the light?

I still think you’re confused about how Twisted works. The tight-coupling you think you see is not real. The limitations are actually strengths. And the elusive, illusive efficiency is real and important to many applications.
Laurent Szyster Says:
May 20th, 2006 at 1:02 am
hmm, let’s see:

> Yep. One uniform interface for one single task. Just like readable() is one uniform interface in Asyncore.

But that readable () is allready called once in the loop as it runs, once in the the async_loop.async_poll function (in Allegra) and *nowhere* else, ever. On the contrary, pauseProducing () is called *everywhere* and *whenever* something, somewhere needs to stall a producer.

> There are quite a few Twisted implementations of pipelined protocols. Nothing about Twisted’s model precludes this feature. It’s quite easy to implement.

Do it. Show me one. That does not call pauseProducing and that can stall output through encapsulation. Something like chunked-encoding for an HTTP/1.1 client and server. Here is one:

http://svn.berlios.de/svnroot/repos/allegra/lib/http_reactor.py

I would love somebody to trample it ;-)

> Take a look at the web2 code. None of it is coupled to FileDescriptor at all.

I did. Want some, here is some:

http://twistedmatrix.com/trac/browser/trunk/twisted/web2/stream.py?rev=16875

go in:

789 class _ProcessStreamerProtocol(protocol.ProcessProtocol):

and more specifically at line 805 of the connectionMade () method

797 def connectionMade(self):
798 p = StreamProducer(self.inputStream)
799 # if the process stopped reading from the input stream,
800 # this is not an error condition, so it oughtn’t result
801 # in a ConnectionLost() from the input stream:
802 p.stopProducing = lambda err=None:
StreamProducer.stopProducing(p, err)
803
804 d = p.beginProducing(self.transport)
805 d.addCallbacks(lambda _: self.transport.closeStdin(),
806 self._inputError)

I’m not sure to understand exactly what this is, but it does not really look like independant from self.transport and heavily rely on defered to work around something. But what? Can you explain exactly what this piece of code is doing?

Sorry to be mean, but remember I’m mean with code, not with you.

Thanks for replying.
Jean-Paul Says:
May 20th, 2006 at 6:01 am
>But that readable () is allready called once in the loop as it runs, once in the the async_loop.async_poll function (in Allegra) and *nowhere* else, ever. On the contrary, pauseProducing () is called *everywhere* and *whenever* something, somewhere needs to stall a producer.

Anywhere you see pauseProducing in Twisted code, an Asyncore-based program would have a pile of custom logic to make sure it stops returning True from readable(). If your application wants to stop reading, that’s just what you have to do. The difference is that in Twisted, there is *one* way to spell this. In an Asyncore program, you have to re-implement it every time you want to say it.

Look at every implementation of producer_stalled in Allegra. There are at least five of these. Each has different, unique application logic. In Twisted, each of those would unnecessary: the application would just call pauseProducing. Instead of implementing a check against protocol state to decide if the producer should currently be paused or not, the protocol can simply call a method.

Take lib/smtp_client.py:108, for example. This is a terrible, terrible way to implement your SMTP client. You are smearing the logic related to recipient addresses all over the reactor and channel classes, instead of isolating it in the code which actually sends RCPT TO commands. This is a clear violation of encapsulation and tightly couples your producer to the rest of your application logic.

In Twisted, there would _be_ no producer during the RCPT TO phase. Once the DATA command had been accepted, a producer would be hooked up to the protocol’s transport and started. There would be no need for it to have any idea whatsoever about the rest of the SMTP conversation.

> Do it. Show me one. That does not call pauseProducing and that can stall output through encapsulation.

I’m not sure what you mean by “stall output through encapsulation”. Encapsulation is a thing which you are not supposed to go through. There’s no reason to want to avoid using pauseProducing (for the reasons given above and in my previous comments), however many of these pipelining protocol implementations don’t actually have to use it anyway:

http://twistedmatrix.com/trac/browser/trunk/twisted/mail/pop3.py#L253
http://twistedmatrix.com/trac/browser/trunk/twisted/mail/imap4.py#L457
http://twistedmatrix.com/trac/browser/trunk/twisted/protocols/ftp.py#L1771
http://twistedmatrix.com/trac/browser/trunk/twisted/web/http.py#L945

> I did. Want some, here is some:

That code isn’t tied to FileDescriptor at all. It will work with any transport that provides IProcessTransport. This is part of the reason protocol/transport separation is so useful. This code will work with any of a number of IProcessTransport implementations: it can even work with implementations which don’t actually talk to a process. This is extremely useful when writing unit tests, since a deterministic implementation can be used to remove the possibility of spurious, environment or load related failures from the test suite. This lets application developers unit test their *protocol* implementation separately from the transport implementation, which may have unrelated bugs, be swapped out with various different versions on different platforms, be replaced entirely by new versions of Twisted, or be provided by a third party who doesn’t contribute directly to Twisted or that particular application developer’s project.

In fact, FileDescriptor itself is an implementation detail. One can hook a protocol up to any kind of transport. Several of the reactors included in Twisted share a core set of implementation modules and re-use FileDescriptor for various means because this reduces code-size. However, not all Twisted reactors use FileDescriptor: take a look at the IOCP reactor, for example.

> but it does not really look like independant from self.transport

That’s right. It relies on the transport attribute of the protocol object, because all protocol objects have a transport object, and they are guaranteed that it will behave in a particular way. What exactly this object is may vary, and this is exactly the reason protocol/transport separation is useful. The protocol doesn’t need to know if it is talking to a TCP socket, an SSL connection, a reliable layer on top of UDP, a child process, its process’s standard output, a unix socket, a FIFO or other pipe, a test fixture, or anything else your heart desires.

> and heavily rely on defered to work around something.

It’s not working around anything. It’s using a Deferred to keep track of the bulk data transfer it has initiated. When the transfer completes, the Deferred fires. When the Deferred fires, the code in question closes the standard input of the process it is talking to. This is necessary to correctly implement a CGI gateway for the web2 server. Note how it does these things in four lines of code which aren’t duplicated anywhere else. These are the four lines necessary to send a stream to a child CGI process. Not four lines plus half a dozen more that are just wasteful code duplication, as seems common throughout Asyncore-based programs.

> Sorry to be mean, but remember I’m mean with code, not with you.

Don’t worry. I don’t think you’re being mean. I think you see some problems in Twisted and you’re trying to make them known. This can be constructive if done in the right way. And I can certainly take criticism of Twisted without being personally offended. However, I don’t think you have actually brought up any valid criticisms. Most of your posts and most of the designs I’ve seen in Allegra seem based on misunderstandings of Twisted’s relatively sophisticated solutions to some important problems in networking.

I think your enthusiasm in this problem space is great, and I think you could make a much greater positive impact on the community if you devoted some of your efforts to contributing to Twisted instead of working on Allegra.
Laurent Szyster Says:
May 20th, 2006 at 6:17 am
>Anywhere you see pauseProducing in Twisted code, an Asyncore-based program would have a pile of custom logic to make sure it stops returning True from readable().

No. Show me one such pile in Allegra.

The readable () method is implemented in a few places. And only in the framework, it will not be implemented in any application. That’s the way to write less code.

For the Async_dispatcher, Async_net and Async_chat class. The UDP_peer may deserve a better one, but TCP client and server channels don’t need anything else. Although async_limit can decorate it (along with the recv and send interface) to meter, timeout on inactivity and throttle channels. With any protocols and transports.

>I think your enthusiasm in this problem space is great,

Thanks.

>and I think you could make a much greater positive impact on the community if you devoted some of your efforts to contributing to Twisted instead of working on Allegra.

I cannot contribute sources to Twisted. I allready forked Medusa. Which was allready not as broken as Twisted in a few key aspects for network application programmers. Twisted broke asyncore and Medusa.

I’d rather continue to use and improve the original.

But I can contribute peer review. And I do.

Regards,
Glyph Lefkowitz Says:
May 20th, 2006 at 8:58 am
> No. Show me one such pile in Allegra.

He did, the various implementations of readable(), which you discarded as “framework” code. You don’t seem to think of it as a pile, because it is in the “framework”. Twisted prefers to avoid repetition everywhere, not just in “user code”.

> Twisted broke asyncore and Medusa.

If you want this to be a productive discussion, please stop using words like “broke” and “wrong” when you cannot substantiate them. I have yet to see a single compelling argument here about a design flaw in Twisted, and you certainly haven’t managed to convince anyone qualified that I respect of your views.

By the way, not only Twisted developers have my respect. I think that the guys who wrote ACE, Zymb, Haboob/Sandstorm/SEDA, various MMO networking engines, and Sam Rushing’s current asynchronous server (sadly closed-source) are all very, very bright. None of them use Twisted. Allegra is not in the same class as any of those projects (many of which are actually highly superior to Twisted in various ways).

You have generally gestured at some areas that need improvement, but you don’t clearly understand what the improvements are.

> I cannot contribute sources to Twisted.

Sure you can. Just port over Allegra to be a Twisted application, rather than a Medusa fork; delet all your mainloop and finalization code, and port it to use Deferreds. :). If you devoted as much energy to understanding Twisted as you did to insulting it, you’d be an expert by now.

It seems as though you are having trouble understanding certain basic concepts in Twisted; however, so I can see why you might not have wanted to use it at first. However, clearly various Twisted developers are impressed by your passion, and would be willing to work through those issues with you until you did understand.

> But I can contribute peer review. And I do.

Eh… don’t do us any favors.

If you think that this peer-review is helping Twisted, you’re wrong. You’ve attracted Twisted developers here to correct your mistakes, not to improve Twisted in response to your criticism. Your reviews are based on misunderstanding the code and mis-estimating its performance considerations. Believe me, I _LOVE_ peer review. I crave it as much as you seem to — it is very hard to get people qualified to comment on Twisted to do so, as most of them are too busy writing massively multiplayer games or switching engines for telecom companies. However, I understand all of your criticisms and they are *not valid*.

In the strictest sense, “peer” review is when qualified equals give commentary. I don’t mean to be insulting when I say this, since you may be very skilled in other areas, and your writing (especially considering English does not appear to be your first language) is quite lucid and interesting: however, you have amply demonstrated that you are not a “peer” of the Twisted team when it comes to network programming and Python application architecture. You could stand to learn a lot by paying closer attention to what has been said here. The reviews JP gave alone are detailed technical analysis that would have cost tens of thousands of consulting dollars to receive.

Therefore, if you are doing this to help the Twisted community improve, please stop. If you’re not going to pay attention to the responses you’ve gotten, you’re just wasting everyone’s time (and perhaps tarnishing Twisted’s reputation in the eyes of those who can’t understand the finer technical points here and just see that “some guy who wrote a networking framework thinks Twisted is no good… maybe it isn’t”).
Laurent Szyster Says:
May 20th, 2006 at 3:50 pm
Hi Glyph,

Thanks for not flaming. Yes I’m just “some guy who wrote a networking framework” and who “thinks Twisted is no good” … as it could be. Not good as Medusa, hence not good as the one I forked from it.

I’m standing on a giant shoulder, itself inspired by an older USENET news server implemented in C with the same design for asynchronous sockets programming.

This design included an IOC defined as the readable () and writable () pair.

That interface *was* replaced by four in Twisted and the IOC was unfortunately removed. Nobody can deny that, nobody did: it is written in the sources, everybody can read this critical “implementation detail”.

So, I’m surprised when you assert that:

>He did, the various implementations of readable(), which you discarded as “framework” code. You don’t seem to think of it as a pile, because it is in the “framework”. Twisted prefers to avoid repetition everywhere, not just in “user code”.

Here I disagree. Let’s count.

Those four Twisted interfaces

stopReading, startReading, stopWriting, startWriting

are all over transports, where Twisted developped a lot. Exactly like pauseProducing and resumeProducing are all over protocols.

Jean Paul counted five modules in Twisted that do implement one of those four calls … outside of twisted.internet. He is missing a lot of the fun. And how many calls are there in each of those five modules. At least twelve. And what about dependencies, all those functions that depend on those calls?

In all Allegra’s async_loop there are only two places where readable () and writable () are called: the two pollster functions (the one for socket.select and the one for socket.poll). Two, in one module.

In all Allegra there are only three documented places where the pair is implemented:

async_core, async_net, ansync_chat

plus one in udp_peer of which I’m not quite sure about (I’m allways learning …) and two pairs of decorators for it in async_limits (soon to be documented …).

Readable () and Writable () have none: they are called in

async_loop.async_poll

directly in the main asynchronous dispatch () loop.

And you will not find in Allegra or its applications calls to something like pauseProducing () or resumeProducing (). That too is taken care of by the two implementations of readable () and writable () that implement buffered I/O for stream transport:

async_net, async_chat

either in the simplest way for netstrings protocols (stall only when buffer are full, stall output only when the queue is empty), or in the most convenient way for encapsulated Internet protocols (async_chat).

More with less code.

That’s the meter for a framework.

I’m sorry to disagree on something that hurts the Twisted projects and applications. But it does hurt. You can deny it for a while, but you cannot deny what is written: all those calls, all those dependencies, all that code.

Look at the sources of:

twisted.web2.http

and look for the part that implements chunked-encoding. Now, try to figure out a way to encapsulated further. Think about Base64 encoded MIME part in an HTTP/1.1 file upload. And that’s just one possible scenario for encapsulation. Think GZIP transfert, charset decoding, etc.

Regards,
Laurent Szyster Says:
May 20th, 2006 at 3:53 pm
Errata, replace “none” by “no dependencies”:

Readable () and Writable () have no dependencies: they are called in

async_loop.async_poll

directly in the main asynchronous dispatch () loop.
Andrew Says:
May 20th, 2006 at 5:25 pm
You seem to be very confused:

“”"Here I disagree. Let’s count.

Those four Twisted interfaces

stopReading, startReading, stopWriting, startWriting

….

Jean Paul counted five modules in Twisted that do implement one of those four calls … outside of twisted.internet. He is missing a lot of the fun. And how many calls are there in each of those five modules. At least twelve. And what about dependencies, all those functions that depend on those calls? “”"

Please check your numbers, as the ones you state here are totally wrong. You can’t have read Jean Paul’s comment very closely if this is the conclusion you manage to draw from it.

Here are the real counts:

Number of implementations of stopReading, startReading, stopWriting, startWriting outside of twisted.internet: Zero.

Number of calls to these FileDescriptor methods in protocol implementations in Twisted: Zero.
Laurent Szyster Says:
May 20th, 2006 at 6:26 pm
To Andrew:

>Number of implementations of stopReading, startReading, stopWriting, startWriting outside of twisted.internet: Zero.

And the dependencies? When some protocol call stopProducing, this method of FileDescriptor will by default looseConnection () therefore check the state of the transport and maybe lose the connection, calling two of three from those four methods.

Do *you* follow dependencies in your count?

This is what I’m tracking through a twisted pile of function calls.

Also, let’s be clear: calling stopReading () is not implementing it.

stopReading () is indeed implemented once.

As a useless articulation. You could as well, everywhere it is called, directly access the FileDescriptor’s reactor and pop the this FileDescriptor from the list to be polled for input.

That’s why it *has* to be called many times, in many places both inside and outside of twisted.internet, and that’ is why it has many obscure dependencies.

>Number of calls to these FileDescriptor methods in protocol implementations in Twisted: Zero.

I found at least one.

http://twistedmatrix.com/trac/browser/trunk/twisted/web2/channel/http.py?rev=16883

at line 316 in the stopProducing method of the HTTPParser class

314 def stopProducing(self):
315 if not self.finishedReading:
316 self.channel.stopProducing()

follow the call to self.channel.stopProducing () it will lead to the

FileDescriptor.stopProducing

which on its turn depends on stopReading () and stopWriting () through loseConnection ().

Do you see the light?

Regards,
Glyph Lefkowitz Says:
May 21st, 2006 at 7:40 am
Laurent,

>>Number of calls to these FileDescriptor methods in protocol implementations in Twisted: Zero.

>I found at least one.

Please read more carefully! What you found is a call to IProducer.stopProducing, an interface method, not FileDescriptor.stopProducing, or potentially ITransport.loseConnection, an interface method, not FileDescriptor.loseConnection. Andrew is correct: there are zero calls.

stopProducing() is an implementation detail. pauseProducing() is implemented in various ways in Twisted (and some outside Twisted), not all of them relying on FileDescriptor.

For example, twisted/internet/_pollingfile.py:_PollableReadPipe implements it by calling ‘deactivate’, not ’stopReading’. This is using a completely different API (the Win32 pipes API, and Twisted timers) and is unrelated to select() state.

Therefore there are already multiple potential implementations which your code could be referring to when calling pauseProducing, and some of them do not necessarily depend on pauseProducing. In fact, it would be breaking encapsulation to even determine that.

This is the whole point of the IoC pattern you’re crowing about: to the extent dependency is possible in Python, Twisted protocols depend on ITransport and IConsumer, *not* FileDescriptor. FileDescriptor is a utility class involved in the implementation of most reactors, but not all of them, and is never visible to the Twisted programmer as a dependency.

Do you understand the difference between interfaces and concrete implementations yet?
Glyph Lefkowitz Says:
May 21st, 2006 at 7:59 am
Laurent,

Since you seem to be missing the point totally here by hyper-focusing on details of code you are not reading properly, let’s try a hypothetical example instead.

readable() and writable() are methods that are called by an event loop. They are not related to the production of data, but instead one particular way which you can implement an interface to an OSes socket layer.

Let’s pretend we have another source of data: digits of π. (This is a hypothetical example, and a more realistic one would be too complex to explain in detail in a blog comment, but I assure you there are numerous applications where I have coded something roughly like this.)

Our hypothetical PiProducer produces digits of π by running a native function in a thread. After producing each digit it buffers it; after accumulating a buffer of a certain size, it delivers the buffer to the dataReceived() method of its protocol.

This data must be delivered to a foreign system, which processes the digits (to what purpose, we might never guess). Sometimes the foreign system becomes overloaded or network congestion interrupts service between the two endpoints. When this happens, the calculation must be paused (the thread must be blocked) until the outgoing buffer can be emptied.

One way to accomplish this is to simply perform the calculation, then write each byte to a blocking socket as it is calculated; I think one of the few things we agree on is that this approach has drawbacks. Another is to deliver notifications (pauseProducing, resumeProducing) to the producer of the data asynchronously, so that it does not outrun its buffer by too wide a margin.

PiProducer is neither readable nor writable, as it is not associated with a file descriptor or otherwise representative of inter-process communication. Nevertheless, it is somehow producing data, and it needs the facility to be paused and re-started. This is true of numerous resources on non-UNIX platforms which are not sockets. Even linux does not allow you to meaningfully check a regular disk file for readability; you have to implement some alternative system for it. You can call pauseProducing on this hypothetical class; you cannot call stopReading on it.

-glyph

(P.S.: You also seem incredibly over-focused on the fact that there are four methods: it could easily have been one method: for example, it might have been called “setActivityMask” and taken 2 optional keyword arguments, “reading” and “writing”: stopReading, startReading, stopWriting, startWriting would then become setActivityMask(reading=False), setActivityMask(reading=True), setActivityMask(writing=False), setActivityMask(writing=True). Or, more likely, _setActivityMask, so as not to obscure the fact that it is an implementation detail and not designed to be called from any code outside the reactor’s implementation. It’s four methods because it makes implementing the reactor easier.)
Glyph Lefkowitz Says:
May 21st, 2006 at 8:17 am
Finally, you keep mentioning dependencies, as if readable() and writable() have none, but Twisted’s approach to this has many.

You’re using a nonstandard definition of “dependency”, as I have mentioned, you are using apparently the exact opposite of the definition that the IoC pattern uses to describe dependency: Twisted code depends on interfaces, not implementations, and therefore the dependency is always one-step deep; what happens underneath the interface is *by definition* an implementation detail.

By your definition, if pauseProducing depends on stopReading, then readable() in allegra depends on async_net_pull (and indeed, all of Async_net) - because async_net_pull sets attributes which one implementation of readable() uses. That, in turn, means async_loop depends on all of Async_net.

I do not believe you think such a dependency exists in Allegra; why do you claim it exists in Twisted, especially given that Twisted uses explicit interfaces to avoid this level of dependency but allegra has only weak conventions and the hope that nothing but sockets will ever produce data?
Chris Siebenmann Says:
May 21st, 2006 at 8:42 am
As a semi-aside, it is well known from a lot of research that anything that wants to scale to lots of connections needs to do as little work per connection as possible. This is one large reason that modern high-connection Unix servers prefer poll() over select() (and often prefer even leaner interfaces); the difference actually matters in practice. Of course, many connections is only one version of ‘high load’, but it is a version that matters for a lot of Internet-facing servers.
Laurent Szyster Says:
May 22nd, 2006 at 6:45 am
To Glyph:

>Therefore there are already multiple potential implementations which your code could be referring to when calling pauseProducing, and some of them do not necessarily depend on pauseProducing. In fact, it would be breaking encapsulation to even determine that.

There are multiple implementations of pauseProducing (), I found one of them which *is* potentially calling stopReading (). What do you want me to do? Look through all that pile of indirections and find each and every way in which, somehow, Twisted sources are still coupled to the event loop?

I don’t have to do that to *prove* that it is the case.

>why do you claim it exists in Twisted

I demonstrated why in your sources. Maybe not clearly, so I’ll try again.

The Twisted FileDescriptor’s “fab four” are adding and removing file descriptors from the I/O event loop. Right? Now, the same event loop is calling the other FileDescriptor interfaces, the ones that implement transport and protocols.

And some of the implementations of those interfaces, not all but many, do call … the “fab four”. Practically, all the ones that must decide wether to include or exclude the sockets from the I/O event pollster.

Here’s is the problem:

I/O event loop -> FileDescriptor -> I/O event loop

There’s an unbroken dependency.

Implementation of what and how to *handle* read and write events must themselves implement inclusion and exclusion from the event loop. You can tolerate that in a single-purpose application.

In asyncore.py, the readable () and writable () interface is *called by* the event loop, all dependencies are broken:

I/O event loop -> dispatcher

The implementation of transport and protocols don’t have to set the state of their dispatchers, they can rely on the one evaluated once *for* the event loop before it polls I/O.

> By your definition, if pauseProducing depends on stopReading, then readable() in allegra depends on async_net_pull (and indeed, all of Async_net) - because async_net_pull sets attributes which one implementation of readable() uses. That, in turn, means async_loop depends on all of Async_net.

But async_loop.async_poll *calls* Async_net.readable () which evaluate a state set by many implementations of the dispatcher’s interface. The effect is radically different.

As for async_net_pull, it does resume collection of the buffer. Readability of the dispatcher will still be evaluated *by* the event loop, *when* it calls.

Doing the contrary is *very* problematic in a framework.

In Twisted, how do you find out in the protocol implementation what is the current state of your FileDescriptor in the event loop? Is it in or is it out, ready for input or stalling output?

How do you know in the implementation of the transport wether the producer is stalling and that you should not put the socket back in the list to poll for output? You must test everywhere. Or maintain state somewhere.

Probably that is why there is a:

producerPaused = 0

attribute set *by default* to the FileDescriptor class. To test, everywhere it will be required, wether the producer paused and flagged or not, then remove or not the socket from the I/O event loop.

That’s a problem Allegra does not suffer from. Because the

producer_stalled ()

interface is called *by* the writable () method. The only places you will ever see producer_stalled () called is in writable (), in a decorator or a boxing producer (like the Composite_producer).

The async_core transports implementations and async_chat protocol implementations “stacked” on a channel do not have to worry wether the producer is stalled or not. And the producers have not to worry wether the channel output buffer is full or not. Or if the socket is still connected. Etc …

Can you see the light?

>allegra has only weak conventions and the hope that nothing but sockets will ever produce data?

Well, if Allegra can be adopted by all the Python network programmers that use sockets *only*, I would be very glad.

Regards,
Glyph Lefkowitz Says:
May 24th, 2006 at 7:23 am
> There are multiple implementations of pauseProducing (), I found one of them which *is* potentially calling stopReading (). What do you want me to do? Look through all that pile of indirections and find each and every way in which, somehow, Twisted sources are still coupled to the event loop?

I want you to understand that an interface is not an implementation, and things implemented in terms of IConsumer and ITransport do not necessarily depend on stopReading, a method not present in either IConsumer nor ITransport. As the interfaces are currently specified, properly implemented Twisted code could survive a full re-implementation of the reactor, removing (start|stop)(Reading|Writing) and it would work *exactly the same*. These are not interfaces used by application programmers, period, the end.

Not only is this the way things are *supposed* to be, this is the way they *are*. I frequently put Twisted code onto different (non-FileDescriptor) transports. Protocols do not ever practically need to call methods outside of their transports’ interfaces.

>Here’s is the problem:
>I/O event loop -> FileDescriptor -> I/O event loop
>There’s an unbroken dependency.

FileDescriptor calls a method on an IReactorFDSet implementor, present as self.reactor. This is a classic inversion of control, or what you are now calling a “broken dependency”. Maybe its methods were called by methods of that very same IReactorFDSet implementor, maybe not. At any rate, FileDescriptor could be totally re-implemented using readable() and writable() (causing a loss of performance) and no application code would have to change. FileDescriptor is an *implementation detail* of a clearly-specified API that does *not* include stopReading and stopWriting.

Here is a simpler example: how does an echo protocol work? Input is received from the event loop, which is passed to a handler, which sends data to the event loop. This is your same “unbroken dependency”, but it is the only possible way that it could work. Sure, you can obscure the dependency by placing your methods in the protocol implementation, forcing your application author to handle buffering themselves, but what good does that do? What application does that enable? What possible value does it have, beyond your aesthetic preference?

To recap, advantages of the Twisted way:

- not polling readable() and writable() is measurably faster and more scalable, even when using standard select()
- a stateful reactor API provides a way to use even more efficient and scalable multiplexing mechanisms like epoll
- a stateful reactor API allows interfacing with existing protocol implementations on top of existing event loops such as libevent, or GUI event loops such as gobject
- explicit interfaces for producing and consuming enable applications which use non-socket sources of data
- explicit interfaces improve testability by allowing you to provide mock implementations
- subprocesses can be treated similarly to network connections by protocol implementations

I have yet to hear a recognizable advantage of the Allegra way.

> In Twisted, how do you find out in the protocol implementation what is the current state of your FileDescriptor in the event loop? Is it in or is it out, ready for input or stalling output?

You don’t have a FileDescriptor. Forget about FileDescriptor. It is an implementation detail. You want to know the state of your transport, which probably provides ITransport, IProducer, and IConsumer.

Is it ready for input? If you want to know, call registerProducer(thingThatWantsToKnow). This will then be sent notifications (pauseProducing, resumeProducing) when the state changes.

You can’t *query* the state of the transport from a protocol, and for good reason. It is only reasonable to respond to changes in this state: if the buffer is full, it is time to stop sending data to it, if the buffer is empty, it is time to start sending data again. The state is hidden, or “encapsulated”, by the implementor.

This is the difference between synchronous polling and event-driven programming. Twisted is event-driven: you are not expected to poll the state of your transport yourself; the framework will *tell* you when it’s time to write. You do not ask, “is it time to write? is it time to write? is it time to write?”, nor does the framework ask you, “is it okay for me to produce data? can I produce data? huh? is it data-producing time? can I produce data?” That is what readable() and writable() do.

Instead, Twisted politely tells you, “Produce some data now, if you would like.”, and “I am too busy to handle more data now, please stop producing it” You may respond to those events as you choose.

A Broken Couple

Breaking everything

Warning

Now is better than never

21 Responses to “Back to the future, in 1996”

adobe writer 9.0

Adobe photoshop freeware adobe photoshop ed free 741.

adobe writer 9.0

download adobe acrobat 9 ed,

download adobe acrobat 9 ed,

adobe premiere elements

download adobe acrobat software

download adobe acrobat software

adobe software multigen v1.0 13 apps

buy adobe professional

download adobe acrobat software

adobe photoshop cs5 portable

discount adobe acrobat

download adobe acrobat software

adobe photoshop free download

adobe writer 9.0

buy cheap software adobe dreamweaver cs5

discount adobe acrobat

adobe premiere database software?

Adobe photoshop freeware adobe photoshop ed free 741.

adobe photoshop cs5 portable

adobe premiere elements

buy cheap software adobe dreamweaver cs5

buy cheap software adobe dreamweaver cs5

buy adobe professional

download adobe acrobat software

buy cheap software adobe dreamweaver cs5

adobe photoshop elements 4.0 for mac

Adobe photoshop freeware adobe photoshop ed free 741.

discount adobe acrobat

difference between adobe reader and acrobat reader;

adobe premiere elements

adobe photoshop elements 4.0 for mac

order downloadable adobe photoshop cs4 extended

download adobe acrobat software

order downloadable adobe photoshop cs4 extended

adobe software lisa beverly?

adobe software lisa beverly?

discount adobe acrobat

order downloadable adobe photoshop cs4 extended

adobe software multigen v1.0 13 apps

download adobe acrobat software

adobe writer 9.0

adobe photoshop cs5 portable

adobe software lisa beverly?

order downloadable adobe photoshop cs4 extended

download adobe acrobat software

discount adobe acrobat

adobe photoshop elements 4.0 for mac

download adobe acrobat software

adobe writer 9.0

buy adobe professional

adobe photoshop cs5 portable

adobe software lisa beverly?

download adobe acrobat 9 ed,

buy cheap software adobe dreamweaver cs5

adobe photoshop elements 4.0 for mac

order downloadable adobe photoshop cs4 extended