In a previous post, I wrote about a situation we encountered on our production Biztalk 2016 system.
After a series of not-Biztalk related issues (including the MSDTC service crashing), we ended up in a situation, where one of our orchestrations would not process any messages. Existing instances would not process, and new instances would only pile up in status “Ready to run”. NONE of them would change to status “Active”, even after restarting the hostinstance bound to that orchestration.
We could not see any hints anywhere (like eventlog on Biztalk- and SQL server, SQL-server locks or similar) as to why the orchestrations would not be processed.
We run a ESB-like architecture, and since that orchestration was part of the main ESB itinerary, no documents could in fact be fully processed.
In the end, we were able to get most the of the orchestration serviceinstances processed, by Suspending all the serviceinstances in status “Ready to Run” and then “Resubmit” them 1 or 2 at a time. If we took more than a few, the situation would lock up again, and we again would have to Suspend all and resubmit them a few at a time.
In the end we had 5 serviceinstances that simply would not process. These 5 were terminated (they were already saved in our Tracking) and after a hostinstance restart (that may not have been necessary), the orchestration would once again process messages without problems.
Another suggestion we had to resolve this situation, was to remove and then recreate all hostinstances for the host which had problems. We did not try this method.
We have sadly seen this problem a few more times.
The reason why all service instances remain in “Ready to Run”, SEEMS to be because 1 or more of the serviceinstances do not contain any messages
All other serviceinstances we have been able to get processed by Suspending them all, and then resuming a few of them. If any of the serviceinstances with no messages are resumed, the problem will return.
Delete any serviceinstances without messages (potentially copy the context first, to assist in finding the original message later), and then the rest of the serviceinstances can be resumes
Were you able to solve it? we are facing exactly same issue right now.
Hi
Yes, we were able to solve it to a certain degree.
In the Microsoft support case we opened, they pointed specifically at two things:
– MDOP (Max degree of parallelism) MUST! be set to 1 on all Biztalk standard databases (including temp-db). In our case, the temp-db was not using MDOP 1
– It must be ensured that enough ports are open for DCOM. In our cases, we had specified a custom port-range in the DCOM setup for 1000 ports. Microsoft suggested that we increased that range – or used default
Apart from that, Microsoft was not very helpfull
We were able to process the backlog of messages, by marking a range and selecting “Resubmit”. However, a few of the backlogged messages were empty, and if we tried to resubmit these messages, the build-up would start again.
Before resubmitting the messages, we performed a restart of the biztalk-server and especially the sql-server (shut down biztalk server hostinstances first)
In general, if you install “Biztalk Health Monitor”, it will point out the two bullets above as well as other potential problems.
Best Regards