Asynchronous non blocking apps
Since a year I'm developing my applications following the usual principle of "non duplication" but pushing it to an higher level then just code duplication.
I'm moving to eliminate the "runtime duplication", or maybe I something that can be called like that.
This lead me to decoupled modules that communicate through messages, of course. This is pretty normal in the world of software integrations, is a well known Integration Pattern. But I wanted to put this into a single, standard application, like a CMS that manage images. There are two way I can achieve this.
Scenario
Let's say that there is a message broker I can use, external to the application, that just works. The app receive a file uploaded by the user. It's an image I need to:
- handle the file upload to file system
- convert file into expected formats
- store file somewhere into the cloud (S3 or such)
- make the image available to the system (associate with database)
Now point 1 is handled by the web framework and is synchronous (but some tricks on the user interface side that are outside the scope of my analysis now)
Simplest solution is to do 2,3 and 4 in a row: that stuck the gui of course. I should have a scheduled thread to search for new files and performed 2, 3, 4 so that the GUI becomes free as long as upload is complete, and the "backend" perform the rest of the processing: there's the scheduled thread to tune and is not that nice, is it?
At the end of point 1, the gui send a light message to the broker saying "there's a new file called ..." and it send this to the right recipient(s). The receiver performs the conversion and send a new message, this time to a new component that perform the upload to S3 and when it's finished send a last message that is received by the original application that performs the association between the stored blob and the database: now the image is available and we can eventually notify the user, maybe with some cool user interface async message.
Push further
Now if the broker is something like ActiveMQ, the components have some listener to different queues or topics and that's done. What is the next question?
Where does those components lives? In the same process of the application? Or maybe somewhere else?It can be both. If they live with the application they can be some spring beans that register a listener on the right topic at startup. But this is a solution that does not satisfy me.
Cause there's duplication, and not enough decoupling. The duplication is that each new app will have to have it's own instances (and configuration) of the same components. Also, if the app is hanged, all those components are and there is no reason for this to be.
Better make them leave somewhere else, and the message broker (and the right queue/topic) is the only link between who procuce a message (say: upload is done!) and the consumer (ok, I'll do perform the conversion!) and so on.
There's a better environment than java?
Now, this is something I've done many times in the last year and I'm trying to make it more "standard" for all my apps. What I'm wiondering is: there a more natural way to do this? I mean, something more deeply integrated with the language?
In java, there are a lot of good tools to do this but everything only exists in my mind and some docs. This kind of architecture really is not supported by the environment, and unit testing that is possible but hard to do an maintain. I'm talking, maybe, about the actor model. Or something that is possible with node.js. Are those technologies really more suitable to create programs the way I just narrated? If so, what is the dark side of the moon?