Random image
User login
Navigation
Why Transports Matter
Mon, 17/09/2007 - 16:25 — sander
This is an open letter to all people related to the content of this article. So, you better read everything to see if that includes you ;-)
Instant messaging clients like Pidgin, Adium, Miranda and Kopete combine support for different instant messaging networks in one program. They often also support platforms that are not supported by the official client software. Therefore, these so-called multiprotocol clients are seen by many as a good thing for interoperability. However, I strongly disagree on this. Instead, my analysis sounds the opposite way:
- Multiprotocol clients are a curse for interoperability.
- Multiprotocol clients hinder adoption of open standards like XMPP and SIMPLE.
- Multiprotocol clients even hinder their own adoption!
- Multiprotocol clients explain why "the Firefox of instant messaging" does not yet exist.
In this article, I will explain these tough claims. Also, I will elaborate about a possible strategy for those who would like to get rid of closed non-interoperable instant messaging networks. In short this strategy is simple: we need to render the walled gardens pointless by destroying the reason they are more profitable than open networks.
Why a Curse and Why Transports?
In this section, I will tell you why multiprotocol clients do not help much to fight closed networks. Not much, as a lot of resources are wasted in these projects which could otherwise be used in a more effective way to get rid of closed networks in favour of open networks. Without closed networks there would be a very friendly and innovative environment suitable for the emerge of "the Firefox of instant messaging".
Ok, but what is the right way to fight closed networks? The right way is a combination of several dedicated XMPP clients with some darn good transports. Transports are server-side gateways that allow XMPP clients to connect to closed networks without any significant effort like reverse engineering or designing a multiprotocol architecture. To be more clear about this abstraction, transports translate the language of the XMPP client to the language of a client on a closed network without the XMPP client knowing that language.
Time for the core of this article...Why transports matter:
-
The quality of support for open standards like XMPP is lower in multiprotocol clients than in dedicated XMPP clients. This can be easily explained by the increased complexity of the software to support multiple protocols, but this is only a minor point that can be solved by having enough eye balls to catch bugs. More important is the fact that XMPP is less used by geeks compared to popular closed protocols. Hence, there are more eye balls looking for bugs in closed protocols support than to catch bugs in XMPP protocol support.
Compare this with instant messaging clients dedicated to the open standard protocol XMPP. First, these clients don't need a complex multiprotocol software architecture. Second, their reason to exist is XMPP support. Thanks to this dedication, the quality of XMPP support is more likely to be much higher than that of unfocused multiprotocol clients. A bug that breaks XMPP support is a major issue in XMPP clients whilst it is a less major issue in multiprotocol clients as these clients will still be usable for other protocols.
Transports matter because they force end users whom mostly use closed networks to contribute their "eye ball capacity" to XMPP support.
-
The 'instant messaging experience'(or innovativeness) of multiprotocol clients is lower than that of the official clients for the closed networks. They run behind because of the required reverse engineering, the complex software architecture, and the tendency to only support features common to the majority of supported popular networks. They tend to limit their feature set to the least common denominator of its constituents, which can be very small. They cannot influence the future of the underlying protocols as these protocols are controlled by the companies owning the closed networks. Where is the room for innovation? There is no room for this as multiprotocol clients simply can never take over the lead of the official clients in designing the future of the closed protocols.
Dedicated XMPP client projects instead, fully focus on a single protocol that is extremely well documented and thus don't needs any reverse engineering. Moreover, XMPP client projects can have their say in the future of the underlying protocols and even can propose their own protocol extensions for standardisation. That's an open door for unlimited innovation!
Transports matter because they allow XMPP client projects to fully focus on innovation without losing interoperability with closed networks and without being influenced or hindered by the closed networks. Transports allow XMPP clients to lead the instant messaging world, instead of following the closed networks owners, like multiprotocol clients tend to do. The later can be compared with Mozilla: no focus, complex, not sexy, following the leader(s), not popular, and so forth. The combination of dedicated XMPP clients and transports could become like Firefox: focused on open standards, innovative, taking over the leadership position, very popular, and so forth.
-
The image of open source multiprotocol clients hurts the broader open source image. Open source multiprotocol clients do not lead in their environment and are seen as poor extracts of the official clients. Unfortunately, this poor image of being a follower instead of a leader means the public will associate open source alternatives by definition with poor extracts of the real thing. Hence, people even may not try XMPP because of earlier unsatisfying experiences with multiprotocol clients.
-
Choice on closed networks is created by multiprotocol clients. Multiprotocol clients allow people who for some reason cannot or do not want to use the official client to connect to closed networks. This choice also reduces the vulnerability of closed networks to malware. You may think these are all good things, but in fact they hinder adoption of open standards like XMPP.
Imagine a world without alternatives to official clients for closed networks. All people who are today contributing to the multiprotocol part of multiprotocol client projects would focus on XMPP and server-side transports. Client projects wouldn't be forced to do urge releases due to some compatibility change in the closed network protocol. Development would be totally independent from the closed network owners. Focus can be directed to innovative client features in the XMPP world. All end users using one of these clients in combination with transports would use XMPP. So, every user would contribute to the maturity XMPP, even if they only want to communicate with people on closed networks!
Transports matter because they can lead to more and higher quality choice in the XMPP world. More choice because the existence of very good transports make it more interesting to create XMPP software. Higher quality because all end users will use the XMPP code, even if they only use their XMPP client to connect to closed networks.
-
Flexibility of multiprotocol clients is extremely low. Multiprotocol client projects are bloated. They need testers for all integrated protocols. The complexity of the multiprotocol architecture needs maintainance. Large changes in the support for one protocol are not easy if this could break support of other protocols, and if they make such changes they have to do this very carefully. Multiprotocol client projects tend to first add features common to all integrated protocols instead of adding innovative features.
Compare multiprotocol client projects with XMPP client projects. Look at the number of innovative protocol features in multiprotocol clients compared to XMPP clients. Also consider how many coders are involved in the different types of projects. The following statistics from ohloh.net about contributors over the entire history of the project can help you. Of course, these numbers are not 100% accurate at all (e.g. some projects give translators commit rights whilst others don't), but still, the significant differences give an indication:
Multiprotocol Clients XMPP Clients Pidgin (33), Kopete (100), Adium (52), Miranda (22) Coccinella (3),Tkabber (2), Psi (7), GOIM (5), Spark (10), Gajim (19), JWChat (3), Bombus (1), Freetalk (4) The fact that there are more active dedicated XMPP clients in the wild than there are active multiprotocol clients says a lot about the required community size to sustain a less complex but focused XMPP client.
Transports matter because they move the maintenance burden and release burden (no forced releases) away from the client projects. They increase the projects' flexibility and development speed (less and shorter testing periods, less complexity). They move away the eventual legal issues in some countries from the client project to a server project which is more suitable to benefit from today's globalised world. A country eventually can forbid multiprotocol clients, but it never can forbid inhabitants to use a transport service hosted in another country without being punished with a trade war by other nations as this would be a trade barrier in services.
The Solution
Multiprotocol clients should become single protocol clients. To accomplish this goal I propose to "steal" community of multiprotocol client projects in favour of XMPP software so that their 'community resources' will become insufficient to sustain a complex design for multiple protocols. We also can ask them to drop support for closed protocols, but I'm afraid they will just laugh at us.
The Tactics
To "steal" community of multiprotocol client projects, he reasons why end users chose multiprotocol clients should be eliminated. Currently, I see two constructive ways to do this:
-
Help the Wine project to support the official clients. In this way, the hard task of reverse engineering and maintaining the proprietary protocol code becomes less urgent for people who contribute to multiprotocol clients, and they may stop doing their hard work. Also, end users will be more happy on their new platform as they can stay using the client they are used to. They can use all features of this closed network...including all misfeatures of the client. If this end user wants choice, he has to install an XMPP client.
Note also that official instant messaging clients for closed networks are strategically an interesting target for the Wine project. Instant messaging clients are probably the most important application of younger people. These people like to experiment with new operating systems. However, they face huge switching costs as they cannot run the instant messaging client their friends use. The switching costs are higher if they need to use an alternative client as they need to learn a new interface for their most important application, and because the alternative multiprotocol client probably does not has all features of the official client.
As a side note, you can see why the Mac OS X version of Windows Live Messenger can't be compared with the Windows version. You also see one of the reasons why Microsoft will not adopt XMPP soon: it hinders youngsters to switch to non-Microsoft platforms. But if the Windows version of Windows Live Messenger can fluently run under any platform thanks to Wine, Microsoft loses a good reason to not adopt XMPP. Note that I do not say they will per se adopt XMPP afterwards!
-
More and better transports are needed to allow XMPP clients to compete effectively with the key feature of multiprotocol clients: interoperability with multiple closed networks. Multiprotocol clients can't innovate as fast as XMPP clients can (see above), so we can tackle them on the interoperability feature which is XMPP's core feature.
Wishlist for Transports
Currently, there is lot's of room for improvement of transports in XMPP land. Because transports are a key feature in the success of Coccinella and other XMPP clients, we would ask the broader XMPP community to improve the current situation regarding transports.
Wishlist for transports:
-
Focus on reliability of service: uptime, hot code update, fail-over, clustering to bypass blocking, and so forth.
-
Focus on reliability of compatibility. Fast deployment of fixes for protocol updates by the closed network owner is what we need. The faster transports are updated, the more expensive and thus the less interesting it becomes for closed network owners to change their protocols to block alternative software.
-
Focus on basic interoperability features only. Text messaging and presence should be the focus. Other features are not important and only make it harder to maintain "reliability of compatibility".
-
More closed networks. Transports should cover all closed networks.
-
XMPP based ideology. Transport projects should favour the XMPP world in the first place. "What XMPP feature will I map to the closed network today?" instead of "What closed network feature will I map to the XMPP world today?" XMPP protocols should be written first and only after that we should look if they can be mapped to closed networks, we should not write XMPP protocols to get similar features as the closed networks have. Taking the lead means not copying features, instead it means innovating with unique protocol features.
Also, transports should be friendly only to people on the XMPP side of the gateway. People on the other side of the gate should get what they have chosen for: no openness, no help, and a bad experience. For example, an error message should sound "You cannot send this file to your contact because your software is not interoperable. Please contact <name of network owner> for more information." instead of "You cannot send this file to your contact because your contact does uses an XMPP client. Please consider switching to XMPP to fix this issue.". When the XMPP user wants to send a file to his contact on the closed network it should sound: "We are sorry but this file cannot be send because your contact uses a chat system that is not interoperable. Please consider to ask your contact to switch to the open XMPP. This issue cannot be fixed without the help of the closed network owner. May we therefore also ask you to sign the petition on <URL>? By supporting this petition, you ask the owner to open its closed network and by this offering a better service to you. This interoperability issue can't be fixed without their help".
As you can see, with a difference in approach we can forward all troublesome interoperability experiences to the real responsible instances (the owner of the closed networks). In this way end users will not associate the issues to XMPP and they may even understand open standards fanboys! This can mean a big thing for the image and popularity of XMPP.
-
No lock-in feature of a server. As is normal in good open standard communities, any lock-in is seen in the XMPP community as evil because it is bad for the emerge of the open standard. Lock-in also can be the root of a software monoculture. We also don't want that. Luckily, server projects are making their components compatible with other servers. And hopefully, this server will follow soon. But we are not there yet.
-
Shaped to die fast. Transports' main goal should be to die as fast as possible. If a transport project dies it may mean the related closed network became open or went offline! Transports should be tools of judo strategy:
The central message of Judo Strategy is that [instances] facing more powerful opponents should avoid head-to-head struggles and other trials of strength. Instead, by employing speed, agility, and creative strategic thinking, they can shape the dynamics of competition in ways that prevent rivals from taking full advantage of their superior strength. At its most effective, judo strategy can even transform an opponent’s apparent advantages into strategic liabilities that undermine his or her ability to compete.
-
Sexy software. Transports should become sexy software projects with a nice website, good documentation, a good community, easy installation and configuration, and so forth. But this item is obvious :-)
About me
In case you want to argue on my thoughts, consider all next facts before you reply:
- I am no real coder; my speciality is copy-paste coding :o) and other ways to contribute to open source projects
- I like open standards
- I like XMPP
- I like unlimited choice in all XMPP software
- I don't think XMPP software is so good it can't be improved
- I don't believe Coccinella is so good it can't be improved
- I also like other XMPP clients; monoculture is evil
- I like successful open source projects like Firefox and Ubuntu because they increase popularity of the broader open source world
- I don't want multiprotocol client projects to die, they just should become dedicated to only one open standard (preferably XMPP)
- I prefer Konqueror over Firefox, but I also like Firefox
Copyright
Comments
Problem is real; transports are a bad solution
You're correct that the need for multi-protocol clients hinders client development. A project like Coccinella would be a huge undertaking if it tried to support all the commercial IM protocols as well as XMPP.
Unfortunately, transports are a horrible solution from a security perspective. They require handing over to some XMPP server your authentication credentials for the commercial IM services. If you run the server yourself, that's fine, but to do that you have to have an always-on machine with a stable DNS address and substantial sysadmin expertise, which does not describe most of the target audience for IM clients. If you don't run the server yourself, you've given away the car keys.
They're also not a very sustainable solution. Commercial IM providers don't like their users using non-approved clients, which means they dislike transports. It's very easy for them to recognize transport servers and block them at the IP level, and we've seen that happen in the past with popular transport servers.
So I'm never likely to recommend that MIT offer transport services on its XMPP servers. We'd be trafficking in passwords we have no business knowing and we'd be subject to being turned off by the commercial providers at any time, with no workaround.
I wish I did have a good solution, other than "hope and hope and hope that XMPP gets more adoption," either from users abandoning the commercial IM services or from the commercial IM services opening up their networks. But I'm sure that transports aren't it.
Best of both worlds
Transports, but installed locally together with the client (or on some trusted pc) and run as a daemon. Yes, there's some overhead and some managing complexity, since users must install one more sw package, but we avoid all the security risks, and in this way transports are seen as multiprotocol clients by legacy networks, with less risks of having their IPs blocked.
I already thought of that.
I already thought of that. Though, the problem is that it is less convenient to setup. Also, the end user needs to update the software himself when the closed network changes protocol. Transports are more friendly to end users. To reduce the risk to get the IPs of the transports blocked, it is IMO better to spread the connections using a clustered transport of which the nodes have different IPs. If you limit the maximum allowed number of connections per IP to let's say 100, the closed network owner may think you are a company using his client. It cannot consistently block IPs with this amount of connections. If they do, they risk to block enterprise users which may deploy their own XMPP server because of this. And that is probably not what the closed network owner wants ;-) So, he will be very careful with blocking, and if not, transports still help XMPP as also "good users" their IP gets blocked ;-)
--
Mvg, Sander Devrieze.
Unfortunately, transports
IMO this is a non-issue:
Why do you think this is easy? I'm not so sure about this. Remember that if they accidentally block the IP of a big company using an approved client for a closed network, this company will lose it's communication channel. I'm pretty sure they will not like this...and they may decide to deploy their own XMPP server because of this ;-) ...another way how transports can help adoption of XMPP. Oh yes, this quote is relevant again: "At its most effective, judo strategy can even transform an opponent’s apparent advantages into strategic liabilities that undermine his or her ability to compete." The opponent's advantage is his control over his network and thus his ability to block access. Undermining is that transports can make them also block the "good users" So, the more the opponent tries to block transports, the more "good users" may get blocked and get pissed about the closed network ;-)
AFAIK they only block IPs with too many connections...so popular transports sometimes get blocked in this way. See also the first wishlist for transports-->"clustering to bypass blocking"
With enough(*) transports in the wild, the closed network owner will spend all his time in trying to block all transports...thus less time left for innovation...
(*) I guess Google and others have an unlimited amount of IPv6 IPs ;-)
--
Mvg, Sander Devrieze.
Nice opinion, but not going to happen
Your ideas are very nice, but very unrealistic. Fact we are missing cool & stable application for XMPP like what firefox is for web is essential. But many things in XMPP is not ideal and we need to offer solid replacement of their proprietary network client before we force them from it.
Understand, what is gateway on XMPP and what is multiprotocol client in nature. XMPP gateway is great service sometimes, but it is very dificult to find good and stable gateways for foreign network. One thing is gateway, that can interconnect with foreign network as server to server. For example, email gateway. I think email gateway is vital for XMPP, as can connect open technologies together. But if you are looking for solid gateway, you will propably find it very limited in features. Biggest problem is of course how to eliminate spam. That can be improved easily.
But gateways connecting somewhere as client are different thing. There are not as many XMPP users as users of popular network like MSN, Yahoo or ICQ. Still, there are too much XMPP users for a little of avaiable and working transports to foreign network. For example, why does not jabber.org host any foreign network transport? It does need resources, it does need work to keep it running. Changes in closed protocol are usual, you need to be prepared for update as soon as possible. Users cannot help you, admin is only one who can fix it. In other words, you need people and resources to provide such transport. Problem arrives when you have big XMPP server with hundreds to thousands users connected at the same time. Then you will have also part of them using foreign transports to keep contact with their friends, who are not yet prepared to switch to XMPP. It is big difference writing single connection client, or 5 connection client. And it is different to write service for hundreds to thousands user online. Not every coder knows how to write such service, even if he is able to make multiprotocol client better. But it is not only about solid transport itself. Foreign networks will block everyone from single IP often, if he is doing too much connections in short time. If you have transport, you have hundreds connection from the same IP. It network does block your IP, you have serious problem, as all hundreds users lost their connection. Then more they try to reconnect, they make more connections and keep it blocked. This is often situation why ICQ transport does not work.
Big difference is also that provider of XMPP service needs to pay for server for foreign networks and for traffic it does generate. They sponsor XMPP to their users for free usually, but why should they sponsor foreign gateways? There might be also legal problems, as almost every network does not permit connection to their network from unofficial clients. Noone will take to court every single user of multiprotocol, but why not sue company providing reliable access to their network for serveral thousands users and has money to get for breaking the license?
Simply, multiprotocol clients are not great, but they does offer to user not primary on XMPP to simply add one icon to their program, and be connected to XMPP. Transports are not always able to fulfill such uptime and reliability, because not everytime it is only their fault. If you are providing service for free for some hundreds users, you would propably like multiprotocol clients, because it does not drain your resource for every single client on foreign network.
I think we are missing good libraries for foreign networks, which could be shared in XMPP gateway and several multiprotocol clients, so it would have better features and better reliability. Almost every client has its own code, which is problem.
I believe we need to make jingle working, before we will be able to get more users to XMPP.
Your ideas are very nice,
Who said adding the missing "cool & stable application for XMPP" is impossible? ;-)
I agree with you that the current transports are not yet ideal. In the wishlist for transports I point out how ideal transport projects look like. What you say is exactly what is in this wishlist for future transports... E.g. who said maintaining and updating a transport can't be made so easy that it's just a matter of apt-get upgrade? ;-)
Every time closed network owners block an IP, they risk to block big companies using their closed network (as they also have many connections). If something like that happens, this company may decide to deploy its internal XMPP server; something the closed network owner probably does not want. ;-) So, if we can make it very risky for the closed network owners to block IPs that may be either transports or otherwise companies, this would be judo strategy in its most effective form (remember the quote "At its most effective, judo strategy can even transform an opponent’s apparent advantages into strategic liabilities that undermine his or her ability to compete."...the advantage is the closed network owner's ability to control the access to his network and so to block IPs) B-)
For the same reason they also provide the XMPP service for free I guess...a better user experience. Also, I had the Google Talk service in mind when I wrote this article...
As already noted in the article, this is the same possible legal problem in some countries that multiprotocol clients face. Though, the risk is less: if some judge in a country decides it is illegal and the software may not be distributed in that country any more, only the transport can't be distributed in that country whereas a whole multiprotocol client would be illegal in that country (which then needs a name change and remove the support for the forbidden network to be legal in that country again)
Only multiprotocol client projects have been pestered by closed network owners so far. Remember how the Gaim/Pidgin project suffered from a legal battle with AOL. With transports and XMPP clients, only transports may be the target for legal actions, the XMPP clients always stay on the safe side and thus development of XMPP clients can't be delayed/hindered by closed network owners.
Take also into account that in several countries the legal status of transports is 100% certain; perfectly legal. So, it will be very hard for closed network owners to take down all servers with transports....much harder than targeting a single multiprotocol client project...
Some hundreds of users is not much...a number transports can easily handle without risking being blocked by closed network owners. There are companies with lot's more concurrent connections on single IP to a closed network...
I heard from coders this would be hard because the libraries of multiprotocol clients are not optimized for lot's of connections. This may be fixed of course.
Already tried the VoIP features of the client of this domain? ;-) (still lots of room for improvement though, like video conferencing support)
--
Mvg, Sander Devrieze.
Well said.
There's just one thing I disagree with you here, and that is that transports should be limited to messages and presence. I think, today, that people have come to expect two more things; File Transfers, and Avatar support.
These two are crucial to get working on transports. It's more or less the baseline of IM nowadays. So, having a transport that have avatar support and a reliable transfer mechanism, like Jingle File Transfers (even if it's only limited to, say, 10 MB large files) is crucial to get adoption going. But, anything more advanced than that...
I can say that in my country MSN is huge. Everyone have an account there, except for a few geeks like me who run Jabber-only. I know that most other countries are the same, just replace MSN with $BIGGEST_IM_NETWORK. And, sadly enough MSN/Live! Messenger is state-of-the-art when it comes to IM and fancy features... :(
Finally, I'd just like to say that you're spot on when it comes to being friendly on the XMPP side and unfriendly on the "other side". If people doesn't know it exists, how can they make the switch?
There's just one thing I
I agree with you on this; a "focus on basic interoperability features" does not mean other feature are not welcome, they just shouldn't be the priority.
I agree that these are nice, but nobody will die without these. Also, IMO it is better to wait until XMPP clients switch to a common avatar protocol (User Avatar using PEP probably) and a common file transfer protocol (I believe this is on the TODO list of the XSF). Oh yes, the item "XMPP based ideology" is valid here: better wait until the XMPP community deployed common and stable protocols before you map them using a transport.
And thanks for feedback (also other "feedback people"; thx :-) )
--
Mvg, Sander Devrieze.
More on transport aggregation
I have a little more insight now into how commercial providers would see transports (specifically AOL; answers may vary for MSN and Yahoo):
* AOL has become more open to the idea of using non-approved clients to communicate with AIM (see developer.aim.com, for instance) as long as they don't perceive a major threat to the market share of their branded client. So AOL is unlikely to shut down your transport simply because they find out it's an XMPP transport.
* However, AOL does do rate-limiting by IP address in order to clamp down on spam and other forms of abuse. If someone starts spamming AOL users through a transport you use, you can expect a service interruption.
* A NAT gateway looks just like a transport to a commercial service provider... and it turns out that NATs with large numbers of users behind them do in fact create problems for the commercial service networks (and vice versa).
So, it's still not really desirable to be aggregating lots of users onto a single IP address via a transport. A transport with a big IP address cluster might have still have issues with spammers.
The idea of transports as locally deployed software has promise, though there are lot of logistical issues to solve there in order to get a friendly user experience.