Workflow Monitoring and Management in Production

[behrad]

How can dataflow be tailored to support a simple flow management. to check if some flows are leaking, blocked or monitor them in production environment?

[apla]

short answer is: there is no management mechanism by design. to my regret there is no general solution for such tasks. in case of simple projects it's not needed, in case of complex projects it's too hard to find a common way to solving management mechanism tasks.

long answer is: for every project we decide how critical are data inside current dataflow (and process). for example, most tasks on web server are atomic and do not have critical data. another case is ftp delivery client (amqp queue => http fetch => ftp delivery). every transaction must rollback when process dies.

if you think your process is persistent or you don't care about workflow data, then you don't need any management. although, for transaction-based systems, best practice is to prepend a task, which creates a dataflow transaction in persistent storage (like mysql or mongo) and after completion wipes transaction data.

1. javascript doesn't have good low level inspector facilities about memory. so, if you have a leak, you have to find it on your own. if you know more than i, please share this information, i would be grateful.
2. if a flow can be stuck or blocked in some way, there is timeout mechanism. as a software architect, you must consider to select between short running tasks and long running tasks.
3. there is no builtin monitoring mechanism.

If i knew your goal better, maybe I could help in solving your issue.