Simple Message Queue Protocol (SMQP) is a publish/subscribe specification that is currently being reviewed by the IETF since September of 2001. Several revisions of the protocol have also been submitted based on people’s comments, and recommendations. This article is to further introduce readers/developers to the protocol and encourage additional input to the specification.Introduction
SMQP is an Internet specification that defines the interaction between a
client and a server application. SMQP defines the communication protocol over a
TCP/IP network connection. SMQP is a text based protocol that is human readable,
though only a software developer would ever see the text protocol in an
implementation. The SMQP specification defines how messages can be passed from a
client, to a server much like how e-mail is passed from an e-mail client and a
e-mail server. The message would then be passed to client’s interest in that
kind of message from the server. Like e-mail, SMQP messages are routed to a
destination client. However, the similarities end there. SMQP is not an e-mail
protocol like SMTP and POP3, but it could be used as one. Like Instant Message
and Chat applications, users who have joined a common “room” can
exchange text. SMQP is not an IM protocol, though it could be used as one. SMQP
is metaphorically similar to these well know Internet applications, but more
extensible and flexible then these protocols. These protocols solved a specific
problem. SMQP solves a more universal problem one of moving messages between
distributed end-points that may not have direct knowledge or connection between
them.
So why another protocol if so many protocols already exist to
perform the tasks that most users are already doing? SMQP was primarily
developed to define an open publish and subscribe protocol that was easy to use,
easy to understand, and extensible. However, in the development of the protocol
and testing it through various use case scenarios and practical implementation,
it became quickly obvious that this one protocol is more than suitable to
perform the same functions as these other protocols, and in some cases better. A
single protocol and implementations of that protocol could replace the
complexity and redundancy of many other well-known protocols that were designed
to solve a single problem. Many of these protocols are like Message Oriented
Middleware (MOM), but are not at the enterprise and abstract level as SMQP. It
is unlikely that these protocols will evolve to a generic MOM specification,
since they are so well established. But since SMQP was introduced in 2001, and
has learned from these other, older protocols, and may be better situated to
become a leading specification for network exchange of information. The author
makes no assertions that this will take place, but once the reader understands
the flexibility of SMQP, they may arrive to the same conclusion.
SMQP uses a centralized distribution architecture where a SMQP
server acts as a message broker for lightweight SMQP compliant client
applications. One server with direct connections to clients defines a single
cluster. Clients within a single cluster may publish messages to any topics
within that cluster, which would then be routed to any subscribers subscribed in
that cluster. The SMQP protocol also supports multi-cluster configuration where
servers can exchange messages with other servers. This allows a fairly limitless
expansions and configuration of the SMQP servers.
The specification (revision 10) has all the details about
SMQP, but the highlights of the SMQP specification includes:
- Centralized, node based distribution
- Publish/Subscribe messaging
- Multiple login instances of a single account
- Point-to-point messaging
- Topic and/or content based subscription
- Message pushing
- Message reply, replace, receipt and recall
- Topic searches, redirects, and access control
- Various notification mechanisms
- Transactional, guaranteed delivery of messages
- Store and Forward of messages (guaranteed and un-guaranteed
delivery) - Text based protocol
- Extensible multi-part, message body with individual
encoding and encryption of data
Extensible authentication methods
Applications of SMQP
This section lists some real world examples in how MOM systems can be used
and how using SMQP would be a benefit for various implementations. This list is
by no means to be exhaustive but provided to stimulate some imaginative
solutions to real world problems. There are various solutions available to solve
many, if not all, of these types of problems. Why would any company invest in
the effort to migrate an existing system to the SMQP architecture and protocol?
If an existing system works and it is not broken, then do not touch it. However,
if an existing system needs to be expanded, replaced, upgraded, or a new
applications needs to be developed and might need to interoperate with other
systems, then a SMQP solution may be correct for you.
- News and information
- List Forums
- Weather
- System Engineering Alerts
- Batch Farms
- Order Fulfillment
- Workflow management
- Customer Relationship Management
- Billing Systems
- Stock Quotes
- Stock Purchases
- Email, IM, Paging, Text Messaging
- Presence of Service
- Data Exchange, file sharing
More Information
More information about the protocol can be found at http://www.omicronsoft.com/smqp
and at the IETF web site at http://www.ietf.org.
Constructive comments can be directed to [email protected].
About the Author
John Tegen is the author of SMQP. He has over 17 years of software
development in the areas of distributing/grid computing, application
development, and user interaction design. He has an undergraduate degree in
Aerospace Engineering from the University of Maryland, is nearly complete with
his MBA degree in technology management, and a long time member of ACM/SIGCHI.
Mr. Tegen was highly engaged in the BeOS community since 1994 and is proprietor
of OmicronSoft. He is currently Vice President of Software Engineering at Stone
Analytics in San Diego, California.
Copyright © 2002-2003. John Tegen
How is this better or worst than XMLRPC or SOAP, or some other system such as that?
Yet another communications protocol?
I have a similar question for John:
How does SMQP compare to MUSCLE? ( http://www.lcscanada.com/muscle/ )
(We hosted a detailed article regarding MUSCLE a few months back btw: http://www.osnews.com/story.php?news_id=1523 )
How is this better or worst than XMLRPC or SOAP, or some other system such as that?
Yet another communications protocol?
After read this article and I have the same question as your.
TAGS and KEYWORDS aren’t really enough; ideally you want to be able to send at least name/value pairs that can be handled in a topic-specific way. I suppose if the topic is Stocks then “IBM” or “MSFT” are fine keywords, but if the topic were computers/x86/desktop/ I might want to have “motherboard:asus” or “floppydrive:no”.
http://www.infoworld.com/article/03/04/18/16imstandards_1.html
I have SOAP for a long time. But the problem with SOAP, XMLRPC is that they are RPC type technologies. And RPC technologies have been problematic from day one! Examples, RPC, DCE-RPC, DCOM, CORBA, etc.
What has worked though are things like MSMQ, MQ Series and other messaging type technologies. Of course messaging technologies are harder to implement in business application terms. But they are more stable, robust, easier to administer, etc…
Interesting I was wondering when this type of technology will come along….
>And RPC technologies have been problematic from day one!
And those problems are ? (except for the bloat overhead of xml in Soap ) ?
SunRPC is imho very nice.
RPC, and especially SOAP is not exactly a real message queue. Yes, it is implemented to have features such as a message queue, and certain B2B systems can be set up with a BUNCH of code to manage guaranteed delivery, asynchronous and synchronous queues, etc, but I can’t realistically connect to a SOAP service the way it currently is understood and BLOCK on reception of a message. And a little opinion from me, XML downright sucks like a bag of ass (I’ve been working with SGML since the latter 80s so don’t tell me I don’t know what I’m talking about).
In SOAP, how could I “block on produce?”
JMS is fantastic because I can use it to talk and load balance and send messages using any vendor. Now on Unix, I have to HOPE that someone implemented SysV IPC and hope that the implementation is not broken (there are instances with SysV IPC where one (or an SA) must obtain root access to clear the queues: not good). MSMQ has it’s oddities as well. Other than JMS, there is no CLEAR standard for accessing MQ handlers. Having it as an RFC, people can now wrap SysV IPC, MSMQ, IBM MQ Series, et all and I shouldn’t care who implements the queue.
For those who don’t work with MQ applications, they are fabulous. You can implement semaphores, critical section coding, load balancing, object pooling, batch & priority processing, anync and sync handling. The most successful piece of software I ever wrote used an MQ as the sole `kernel’ as a message passer between business object producers and consumers: a very different way to look at an application, but more powerful than people realize.
—
3lixyqueue
YAIS (yet another internet standard). I have started to understand why people go to windows. One standard. A shitty one perhaps, but still. One standard from one company. Trying to keep up with all the standards is a pain in the ass.
Maybee I’m just kranky this morning….
Developing systems based on MOM takes a different mindset than using RPC. Many of the features you expect from an enterprise system such as guaranteed once delivery, 100% uptime, and high scalability are “free” using a product like MQSeries. However, until JMS came along, there wasn’t a high-quality yet inexpensive implementation for most developers to run against.
However, most of the affordable JMS implementations can’t hold a candle to MQ (Orion comes to mind). Here’s hoping that SMQP spurs some innovation in the inexpensive, commercial sector.
A message passing protocl, with a lack of flexibility in some spots where it would be nice to be flexible, and a lack of rigidity where it’d be nice to know what to expect.<p>
This looks like a bad reimplementation of XMLRPC/SOAP/et al. Like everyone else, I just can’t see how this brings any benefits, since it doesn’t seem to actually define any queueing behavior that would add onto existing message passing human readable protocols.<p>
If the author wanted to do something useful they’d write an MQ architecture for *nix that was open source and freely available and that was message passing protocol independent.
Greetings:
Thank you for the comments so far.
SQMP is a message queue protocol that allows not only point-to-point transmission of messages, but topic and content based. It is to be more than a simple IM protocol. SMQP has greater authentication, guaranteed delivery (transactional), multi-part encryption and compression.
SMQP is a MQ protocol much like proprietary MSMQ and MQSeries (WebSphere MQ), but open. It has many of the same features found in MSMQ and MQSeries.
JMS is a java interface specification for messaging services, not an implementation. JMS could be implemented using SMQP.
SMQP is a non-blocking protocol; send and continue much like email. RPC systems are a call and block until remote procedure is complete. SMQP would be a request, async notification (though you can sometimes fake an async RPC request).
As someone mentioned earlier, SOAP is not a MQ. SOAP/.Net is still struggeling with how to manage a real MQ system for web services that is not just a point-to-point request (SMTP). Additionally, SMQP is one protocol, not a combination of syntax like HTTP, XML, WSDL, and UDDI.
Thank you for noticing that. It has been coing back and forth to have just named tags or a meta desciption of n/v pairs. However, the largest con against meta fields is that the other nodes would need to know the context of the named fields. For example,
SYMBOL=IBM
Then subscribers would have to know these names exists. The TAG approach allows it to be more free form much like a query at a search engine or an HTML meta header definition. The specification allows for begin, contain, end, and approximate string matching.
As the protocol get more widely used, meta fields can be looked at again as long as the context of the names could be universally understood.
I have started to understand why people go to windows. One standard. A shitty one perhaps, but still. One standard from one company.
The official standard
http://www.iana.org/assignments/port-numbers
Thank God, we’re using one standard from many companies and universities instead of MS standards.
Greetings.
I guess I need to know what you mean by ‘queing behavior’.
In terms of topics which is basically a message queue with a hierarchial context, it has the following controls:
* User access control (read/write)
* Delivery control to subscribers (all, one, or none)
* Priority control
* Auto receipt request (by server not subscriber)
* Message reply, recall, and replace
* Timeout control when subscribers are not present
* Sequencing control (messaging ordering)
* Message maintenance (delete, move, count, list, etc.)
* Subscription windows (since and/or until date/time)
* Subscription suspend and resume w/o unsubscribing
* Topic property change notification
* More…
If you have other needs to govern queues/topics that is not mentioned here or in the specification, I would like to discuss them more with you. As the posting mentioned, we are looking for additional features that are needed by developers trying to implement a MQ system and to improved the depth the SMQP specification.
I’ll read everything Monday, but does this spec allow or consider bounded queues?
—
3lixyqueue
John, you seem to know the topic pretty well. You’re involved in IETF activities?
And FYI, you might catch some bad waves by regular trolls pretty soon, I hope you’re aware of it.
Revision 11 will have properties on a topic that can place limits (boundaries). The currently planned topic limits include:
Number of subscribers
Number of messages at a given instance
Maximum individual message size
Maximum size of all current messages
A return code of 414 is returned when a resource has been exceeded. Additionallu, a server implementation can throw a 414 error code if other resources have been exceeded that are outside the protocol.
I am the author of the SMQP that has been in work for a couple of years. I’ve been working with the IETF as a specification author but not as a member of the task force. SMQP is an Internet-Draft. It is up to the IETF to place it on the standards track. This takes time. I’ve been getting excellent feedback from people over the last two years to make SMQP an enterprise level solution.
I’am aware that some will not receive this positively. I hope that they can become knowledgeable in MQ solutions and/or have used MQ solutions to understand the importance of having an open specification to manage this.
John wrote:
As someone mentioned earlier, SOAP is not a MQ. SOAP/.Net is still struggeling with how to manage a real MQ system for web services that is not just a point-to-point request (SMTP). Additionally, SMQP is one protocol, not a combination of syntax like HTTP, XML, WSDL, and UDDI.
Having read the spec (cool!), I would have to differ here on a few points. First, SOAP (at least recent versions) is really just a message formatting standard. Any message transport that is specified is just given as an example. SOAP does not include WSDL or UDDI – those are seperate specifications built on SOAP. For that matter, SOAP isn’t necessarily a “web service” thing either.
SMQP is a transport protocol, more comparable to HTTP or SMTP than SOAP. Like those protocols, it doesn’t specify the format of the messages it carries.
Given all that, I don’t see why SOAP and SMQP should be seen as competing at all. Actually, I think that they would work together rather well, using SMQP as a transport for SOAP messages. In many ways this is similar to what the Jabber group are doing with their protocol.
What I’d like to see is a comparison between SMQP and Jabber. They seem to be tackling the same problem domain in quite similar ways, if you ignore syntax differences.
Agreed with Brians Hawley’s comments. SMQP can easily be used with SOAP. Jabber is moving in that direction, but we did not want to be so XML based. We wanted the commands to be tight and light-weight. XML is a very verbose syntax which is both good and bad.
Jabber is more competing with APEX that is using the BEEP specification (RFC 3080). But both APEX and Jabber have yet to deal with some of the issues SMQP had to deal with earlier in its development. SMQP dealt with security, authentication, and encryption early in its development for one.
It can be confusing with many seamingly competing protocols. In the end, its not the protocol that is the best that ‘wins’, its the protocol that is adopted and used the most.
Given the simplicity of the syntax, it shouldn’t be difficult to implement SMQP client code. Are there any patents that would prevent me from doing so? How about server code?