Tuesday, November 29, 2011

Amazon Simple Queue Service (SQS) - Simple in good and bad times

Now I am biased having just dealt with IBM's MQ product line and the walled garden approach to their tool suite, so I might be over complementary to Amazon AWS SQS.  Okay, one more thing about IBM MQ, just look at the website, tired, old, selfware; some might argue stable, but hype is the name of the game.  On the other hand AWS SQS fresh and simple, by name, approach, use, and in features.  However, I believe there is something simply excellent about that.

Simple Setup - I had a queue up and running in 30 seconds.  I clicked on the AWS Management Console tab, a browser/java based tool and entered some stuff in and bam queue.  Then I right clicked on it, snappy, I can write a message to it.  So, again I right clicked on the queue entry listed in the AWS SQS console and found a real-time queue reader/listener, and hey I can even delete messages.

Simple Developer Tools - Sure enough I flip over to the eclipse based tools and as expected I could access the queues from there as well.  Nice I can write my 64KB per message max from eclipse, integrate with a plug-in, create a new queue, or whatever.

Simply Service Enable - Most important to me is that every feature is web service enabled.  So, anything Amazon SQS has not automated or simplified can be further refined and utilities written.  To some this may see interesting or novel, but let me assure you that large companies are slowed by walled gardens and management consoles that only the privileged or support companies have access to.  If I dynamically want to create a queue for every batch so I can keep track of them, fine.  If I want a queue for each and every of the million customers, fine, go ahead.  The simple way to put it is that life cycle and direct control matters in large organizations period.  Want to integration just an HTTP call with some XML error handling, which of-course is documented and has a schema, see for yourself, SQS API Documentation.

Simple Documentation - The documentation is short and simple.  I understood how the main drawback or assumption AWS SQS was built on within 5 minutes.  The examples drove every question I had, only so many you can have when you can list the functions on two hands.  Read the architecture document, here.

Simply a message - I can't knock IBM here either, but this is about AWS SQS.  A message can be any text up to 64 KB and can live for up to 14 days.  I have spend weeks trying to wrestle simple non-functional requirements or constraints from product vendors or existing deployed environments.  Not much to question when it is documented and any specific queue can be checked via the management console or a simple service call.

Simply Assumes Idempotent - Which is smartly pants for safe from replay attacks of messages due to outages or glitches.  They chose the word Idempotent and they chose to saddle the consumers of the queue with this to ensure simplified recovery of nodes in their queue clusters.  Now don't get me wrong as designing a consuming system to be Idempotent is not extremely hard, but it pushes the process and data locking onto your application performance.  So you get fast queues, but certain transactions or types of applications will be hard to write.  For example updating a Facebook status or tweeting twice might be fine, or an Order system that commits a transaction, then it is easy for an consuming application to be Idempotent.  However, what about a complex relational multi-transaction rules engine driven process?  Not so easy now.  Now because I cannot trust the AWS SQS to not make a mistake the consuming application could have an architecture performance flaw, which could be avoided by a more rich queue management capability.

Simply distributed and centralized - There is a constant tension in an Enterprise to centralized or disperse.  This is very true for enterprises with countless systems.  AWS SQS is hosted at Amazon, period, and cannot even be easily distributed.  If that data center disappears there could be issues.  If an application wants to use it for internal communication without having to leave the network or memory, AWS SQS will let them down.

Simple Summary - So simplicity is a double edged sword that can provide architectural nirvana or nightmares.  Don't get me wrong I am a huge fan of Amazon's approach to creating service oriented resources for the masses.  They even give you thousands of free messages a year which inspires me to learn more and develop demos with SQS, but I cannot use it for real enterprise demands; darn.

Sunday, January 10, 2010

UML supports the MDD process: No match made in heaven or oxymoron.

This post builds on previous posts (1,2) and is serves as additional commentary and context to a post by  Franco Civello - UML for MDD - Oxymoron or match made in heaven?  Not sure what Model-Driven Development (MDD) is exactly, then check out IBM's MDD Redbook which is one perspective.

UML is a modeling language with thousands of features and facets.  Most people discount UML, even experienced modelers and language developers, without understanding what the language is capable off.  The primary UML feature missed of importance here is the UML Profiles capability.  People miss that UML Profiles allow for extension and constraints of the language itself, read section 18 of the UML Superstructure or take a look at the UML profile catalog.  The two linked posts (1,2) speak to what UML profiles are etc, so you might consider reading them before proceeding.  This extension and constraint mechanism is exactly what is needed to create domain specific models i.e. support MDD.

Now MDD is just a process, so what are the actual implementations or tooling used?  The following are just examples and it is not complete as I am sure there are a hundred ways to do it.  Firstly, there are DSL (Domain Specific Language) tool kits.  A classic example would be YACC and LEX, but a modern example is Eclipse XText Model/Language Definition.  Another significant way is with DSM (Domain Specific Modeling) tool kits, such as Eclipse EMF or MetaEdit+ (a leader in the DSM space).  The DSM space is not free from it's challenges too, but they are further along on average.  Think of DSMs as graphically represented DSLs if you are newer to this area.  Finally UML Profiles are another way to implement the domain specific tooling required.  Tools such as Rational Software Modeler (RSA/RSM), Eclipse MDT, and other tools.

What adds to the confusion between DSL/DSM and UML Profiles when it comes to the creation of domain models for MDD processes is approach and familiarity. The good DSL/DSM tools typically are tabla rasa (blank slate) model creation, much like a empty file when you start programming.  There is less baggage by starting with a blank slate, you also do not have to worry about removing or limiting existing model elements.  Now strictly speaking for UML Profiles you can start with a basically blank slate, but most tools don't support this.  A modeler in the UML Profile space must not only construct/extend the model, but constrain the other model elements that are not needed or conflict with the DSM.  You also have to research and pick the right UML element type to extend to minimize the constraining work you have to do.  Interestingly, in my experience creating small DSMs/logical constructs for a group of architects having a rich language actually reduces the work, but this need not always be the case.

You could really do MDD with any of the tools above, the question often becomes what is the cost, time, and skill level needed by the resources?  What actually seperates the tools are the features and ease of rolling out model tooling to all of the involved parties, such as business, vendors, developers, etc.  So the seven hundred pound gorilla in the room is actually the lack of wide spread usable tooling to build UML Profiles.  Few tools if any implement all of the UML Profile features and creating tooling for use in the MDD process can be very difficult.  I suspect most people use free or drawing oriented UML tools and thus never get exposure to UML Profiles.  Even if they did they would have to buy a specific software suite to support it fulling.  I use Rational Software Modeler (RSM/RSA) which is thousands of dollars and even it does not correctly implement UML Profiles, but it is close.  The DSM tools are not all free, but at least there is Eclipse MDT with EMF/GMF and the generative tools needed.  In a year or two Eclipse may have an offering that is usable enough for novices and support UML Profiles for the tooling efforts more fully.

So UML and MDD go together just fine when you understand the extension, constraining, and profiling capabilities of the language, but all is not well with the union.  UML requires lots of time and energy to learn and understand it before it will begin yielding results.  Better cheap tooling and tutorials are still needed to support the modeling and programming communities at large lest UML/MDD/MDA remain for 'experts only' which I would quantify as a failure by it's advocates.  Much of these challenges are not just limited to a UML Profiles based approach to MDD.

It is worth noting that at some point the industry should look back to Case Tools and YACC/Lex to evaluate why Domain Specific movements have failed in the past, or we may just repeat it.

Monday, January 4, 2010

Reference Cards, Cheat Sheets, Posters, whatever they are called.

Tonight I tried to find quick reference sheets for both UML and BPMN, free of course.  There are quite a fewer different terms for them be it poster, reference cards, cheat sheets, quick guides etc.  However, once I got into several sites by searching for "Reference Cards" I found the other names for these small printable guides to various technologies. Oh yes, and much more managable or likely to be used than "pocket references" which don't fit in your pocket.

List of quality sites quick guides:
refcardz.dzone.com
itPoster.net (singular)
cheat-sheets.org
scribd.com/(kind of)

I did not find a suitable quality UML guide (Stackoverflow has a list), but the BPMN poster at itPoster.net though outdated at time of this post was still really well done. There are several red herring links which get referenced a lot, but are not that good, ex: (How are you to carry this around?) Any other good sites that have a listing of printable reference guides?

Friday, January 1, 2010

Rational & Eclipse UML2 Tools - Flawed UML Component Connections

Connecting UML Components with the Ball and Socket notation in Rational Software Modeler (RSM/RSA) and Eclipse UML2 Tools is not fully implemented or flawed. Too fully understand my point you have to keep in mind that these tools are for modeling, not just drawing a picture/diagram. Thus the internal model representation of what you diagram must be correct and complete.

The Need:
I discovered this issue while trying to design a suitable domain specific language/model for architects to talk about systems and what interfaces/data connections they had between them. The users understand components, so our systems were components. Additionally, interfaces at a high level make sense for being an "agreed upon" contract of interaction between systems. So, the components will realize the interfaces or <> them to show providing and consumption between the systems. The figure below shows two examples of this.


If you look closely at the diagram you will note that the ball and socket do not really connect, which is the first indication that there is a problem. RSM will not let you connect the ball and socket visually or in the model. The Eclipse UML2 Tools will let you connect them, but it fails to model them correctly, though it is still in the early phases of open source development so it kind of gets a pass for most of this discussion.

UML Defines:
So what does UML say should be possible, both from a visual representation (syntax) and model structure (more semantics)? The UML specification 2.2 (Superstructure) says on pg 151 in section 8.1 at a high level that:
"As a result, components and subsystems can be flexibly reused and replaced by
connecting (“wiring”) them together via their provided and required interfaces....
In addition, the BasicComponents package defines specialized connectors for ‘wiring’ components together based on interface compatibility."
The Assembly kind of Connector
At the syntax and semantics level however there are a few details to understand. The major construct to support "wiring" are Connectors. The ComponentBase extends the Connector definition allowing for two kinds of Component specific Connectors, Delegation and Assembly. Also, you "wire" both components and component parts/ports which can make the spec hard to read. It states on page 155 in section 8.3.3:
"The connector concept is extended in the Com ponents package to include interface based constraints and notation.

A delegation connector is a connector that links the external contract of a component (as specified by its ports) to the internal realization of that behavior by the component’s parts. It represents the forwarding of signals (operation requests and events): a signal that arrives at a port that has a delegation connector to a part or to another port will be passed on to that target for handling.

An assembly connector is a connector between two components that defines that one component provides the services that another component requires. An assembly connector is a connector that is defined from a required interface or port to a provided interface or port."
Notice that for assembly it does not say compo nent part, but components and uses the word between. To make things confusing the major examples (pg 153 Figure 8.15) are on Component Instances that are Parts, not between components so to be sure I also found the following on page 156 under constraints:
"[5] An assembly connector must only be defined from a required Interface or Ports to a provided Interface or Port."
An interesting note, but I will go no deeper is that the table 8.2 makes it sound like connecting the ball and socket by a usage/dependency would be equivalent, especially when you factor in
Figure 8.14. Table 8.2 from page 160:
So, the assembly connector defined in UML supports connecting the ball and socket between components, which should be reflected in the model as an Assembly kind of Connector.

Ecore/EMF Provides:
It turns out that the Ecore/EMF model of UML which Rational and Eclipse UML2 Tools leverage support the Assembly Connector per the UML spec. The below two figures show from Eclipse the construction of a component required interface connecting via an Assembly Connector to the same interface provided/realized by a component. The properties could be set and the selector filtered the options correctly given that the interface type had to match.


How Rational (RSM) FAILs gracefully:
In Rational you cannot even create an assembly connector between two components either visually or in the model. The diagram canvas will not allow you to select a component that is not an instance nor will it let you specify it when you are drawing a new (yet to be typed) line. In the model if you try to add a new relationship between two components it will not give you the assembly connector option.

Even worse if you try to connect the ball and socket in Rational with a use or depends you will actually be connecting the interface to itself, see below. Basically the ball and the socket are representations of the interface, not the provided or required ones.


Sadly, Rational provides the Assembly connector and even renders it fine when you are using instances of components (Parts) in the structural view inside a component. So, in my estimation the just mis-understood the specification or they did not keep up with the changes to UML as this was added as part of 2.0 from what I can tell.

A bug will be logged with IBM for the Rational Software Modeler (RSA/RSM) tools, and the ID/Link to it is TDB.

UML 2.0 - Pocket Reference - If you really are going to carry it around


I have to admit that I am lured into the pocket size form factor used by O'Reilly and others. The cute baby animals on the cover, like the gorilla on the front of the "UML 2.0 - Pocket Reference" is just fun. I also have "The Elements of UML Style" in the same easy to consume and not really pocket sized size. Parts of this review also apply to both pocket books, but the majority is about the UML 2.0 - Pocket Reference.

If you do actually end up bringing this "pocket" reference around with you then great, but consider when will you actually use it. While you are walking around the park thinking about UML and all of its confusing details? This book only covers syntax and usage, not really semantics which is where most people struggle anyhow, as the semantics and detailed syntax consume over 1000 pages. You could bring it to meetings and wow people with perfect diagrams. Again, this book might help with that, but how often are you drawing a diagram on the white board and everyone is confused because you lines are not specific enough, or the shape is just not right. This does not happen because the people in the meeting have the context of the conversation and thus do not need perfection, the drawing is just a tool to move the conversation forward not get hung up on syntax, semantic maybe, but this book won't help with that beyond the bare surface.

Additionally this spartan "pocket" reference decided to include OCL, which is 5 pages long. Distilling a language, context, and usage in 5 tiny pocket pages really was just a waste of space. There are 100+ page books on OCL to do it justice, if the book is even current or matches your implementation of UML.

It seems far more rational to make your own pocket guide using a print out from websites with examples or the front and back flaps of UML Distilled, by Martin Fowler. This follows the 80/20 approach in that you get 80% of the value with 20% of the cost and complexity. Pick the diagrams or elements you typically use, such as classes and components or sequence diagrams and limit your pocket guide to those. Or just bring your laptop and have some favorite UML sites bookmarked.

Finally if they are really trying to make a reference, create tabs for the sections at least, having to look up the page in the front or term in the back is not very efficient if you use the book frequently. And if you are like me and the pocket references end up on a bookshelf they don't even fit in, the black sheep of your library.

Buried in the depths of Eclipse Team (Sync), it can support UML Models after all.

The short lesson: When looking at eclipse features dig deep and work off the website and examples. Books are outdated quickly and are often written to just the major usage of a package or feature. Never be to proud to use the eclipse project specific mailing list.

The story: Whenever I take time off work, I invariably take on creating some new thing or read a technical book, basically just go down a rabbit hole of some kind as much of my day to day work requires abstraction from the details and hands on coding.

The Christmas 2009 break is no different. This time I dug a little deeper than I normally do with eclipse and plug-ins. I have been creating an eclipse plug-in to import UML objects from an outside server which houses use cases, requirements, and test cases. I figure why copy and paste or manually sync them which you can have a wizard done. Great so after a COM Bridge, EMF Transactions, UML Profile, and a shovel I have an import working. I did not do this in a vacuum, but instead have been going off, eclipse books, the eclipse.org dev websites etc. I had checked into Eclipse Team about 8 months back and found it was decidedly IResource (File/Folder/Project/WorkspaceRoot) limited as all of the methods seem to require IResource to operate. This lead me to just drop the "Syncronization" and Team approach to integrating UML models with outside data sources. Heck I figure even IBMs Rational suite has to do all the sub-file (UML object) comparisons with a comparitor, that must be it.

As I completed a workable Alpha of the importing plug-in and began to lust after the export, update, and delta features one expects when collaborating, I decided to double check that the Eclipse Team (CVS,SVN, etc) really would not work. Into the rabbit hole I went. This time as I went to the website and played with examples. They all showed promise as features had been added in recent releases. The org.eclipse.team.examples code is helpful for understanding this seriously complex package/feature. But more importantly there had been additions since 3.2 and how there seemed to be support for Logical Models. Great models are what I am doing.

So I really dig in. The trouble was that I cannot find a good example to how this would be done in a non trivial way. The example Logical Model code was trivial as each "Model Element" was still a IResource, so it did not show that this could be done. Even though they had in the API/Eclipse documentation mentioned Ecore models it all seemed to be still focused on IResources. Finally, however, I am convinced that the Logical Model Team feature set will meet the need of Syncing non-IResource model components, ie sub-file detail level representation. The wiki page about the team logical model objective said it most clearly.

The following paragraph, taken from the Eclipse 3.2 plan, describes the motivation for the logical model integration support that was added in Eclipse 3.2.

The Eclipse Platform supports a strong physical view of projects, files, and folders in the workspace. However, there are many situations where a physical view is not the most salient or useful for many purposes. In some cases, multiple distinct objects happen to be stored in a single file, like an archive. Conversely, in other cases, something that is logically a single object is stored across multiple files. This discrepancy between logical and physical creates problems for common operations such as searching, comparing, and versioning, which need to work in the physical realm. Eclipse should support some way of mapping between a logical view and the physical organization of files on disk.
At the beginning of the 3.2 development cycle, we captured the requirements from several clients in a requirements document.


And while I was writing this post I found THE emf example that has not been merged in with the other org.eclipse.team.examples, seriously, I can't even find an email dev list to post about this issue. Also, the link on that page is dead, I found the code in the CVS archive however.

So, hours of playing with the examples, running them in the debugger trying to figure out the slight of hand and object types being used, and reading pages of seemingly contradictory documentation, nearly giving up assuming that it must not work, only to search or dig just a little bit more and find out it is supported. I know way more about the Team feature set now then I actually need to know for writing the plug-in. Hopefully next time I will navigate the eclipse web a little more efficiently, or I am find a mailing list and enlist the help of others rather than just digging deeper into my rabbit hole alone, and even Google Search failed me.

Monday, September 28, 2009

Every Software Architect's Dream - Whiteboards everywhere

I like most people, I suspect, especially software architects and programmers, dream of white boards on every wall floor to ceiling. I have wanted this for some time for my office so I could doodle hours away in my office getting dizzy on the dry erase fumes. In the past I had looked for a quality product and failed, until now. Visiting my undergraduate college (St. Olaf) the computer science for it's brand new shared building has for entire rooms painted a white board surface onto the walls. STUNNING! Want to convey an idea, design a group project, doodle, noodle, or whatever, just walk up to a wall. I hope it holds up and I hope to get around to painting my office soon. I think they are using one of the more expensive products, IdeaBoard, but I am emailing the school to double check.

Finally a canvas large enough to hold all of the crazy UML diagrams and brainstorms.