OSB 11g: Stuck Threads when using inbound database adapter

Using a polling database adapter in a OSB proxy service, one may have noticed the following behaviour in Weblogic server:

  • an exception in the server logs about one or even more stuck threads like this:

<BEA-000337> <[STUCK] ExecuteThread: ’10‘ for queue: ‚weblogic.kernel.Default (self-tuning)‘ has been busy for „705“ seconds working on the request „weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl@21b9db0“, which is more than the configured time (StuckThreadMaxTime) of „600“ seconds. Stack trace:
Thread-118 „[STUCK] ExecuteThread: ’10‘ for queue: ‚weblogic.kernel.Default (self-tuning)'“ <alive, suspended, waiting, priority=1, DAEMON> {
— Waiting for notification on: oracle.tip.adapter.db.InboundWork@21b8e86[fat lock]
java.lang.Object.wait(Object.java:???)
oracle.tip.adapter.db.InboundWork.run(InboundWork.java:498)
oracle.tip.adapter.db.inbound.InboundWorkWrapper.run(InboundWorkWrapper.java:43)
weblogic.work.ContextWrap.run(ContextWrap.java:39)
weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:203)
weblogic.work.ExecuteThread.run(ExecuteThread.java:170)
}>

  • and/or server health state from OSB managed server changing from state „Ok“ to state „Warning“

Such a behaviour alerts administrators, thinking that something is wrong with the deployed applications or OSB services.

Looking in the Oracle documentation one can find the information that this is behaviour by design and that it can be ignored. To verify the OSB proxy service’s database adapter as the source for this the proxy service has  to be simply disabled in OSB console. Doing so makes the stuck threads disappear. The behaviour seems strange at the  – so why this?

When defining an inbound database adapter, Weblogic threads are used to perform the polling on events occurring in the defined database. Because OSB is designed to deliver high performance and throughput, a number of threads, which depends on the numberOfThreads property in the adapter’s JCA file, is exclusively reserved for the database adapter to perform the inbound polling. Such reserved threads will, due to performance reasons, never be released and never be returned to the corresponding thread pool. After the configured thread timeout, which is by default about 600 seconds, the stuck thread behaviour occurs.

Although this is the default behaviour, it is really confusing and could lead into serious problems, if real threading problems occur in deployed applications/services that are not caused by the adapter threads and which will not be noticed and handled in time. So what can be done to get rid of the stuck adapter threads?

The Oracle documentation proposes to define a separate Work manager and configure this with the proxy service transport’s dispatch policy. To do so, the following steps has to be performed:

  • Define a custom global Work manager using weblogic console with the OSB managed server as deployment target


work_manager

  • Configure the new defined Work manager to ignore stuck threads

ignoreStuckThreads_wm_config

  • Configure OSB proxy service transport’s dispatch policy to use the new defined Work manager

OSB_proxy config

Afterwards the stuck threads behaviour caused the OSB proxy service or by its configured inbound database adapter should not show up again.

Über svenbernhardt

Sven Bernhardt is a leading SOA/BPM architect and works as a solution architect for OPITZ CONSULTING Deutschland GmbH―a German Oracle Platinum Partner. In his role, he follows his passion for designing and building future-oriented, robust enterprise applications based on pioneering technologies. Sven is involved in diverse, large SOA and BPM implementations, dealing with challenges in the areas of business process automation and enterprise application integration. He also has longtime experience as an SOA/BPM coach, trainer, developer, and architect. Sven is an Oracle ACE and a frequent speaker at numerous IT conferences.
Dieser Beitrag wurde unter BPM & System Integration abgelegt und mit verschlagwortet. Setze ein Lesezeichen auf den Permalink.

2 Antworten zu OSB 11g: Stuck Threads when using inbound database adapter

  1. narendar schreibt:

    hi,

    I am dealing with the same issue, but in my case i am unable to poll the records, can you help me with this?

    Thanks
    Reddy

    • svenbernhardt schreibt:

      Hi Reddy,

      what is your concrete use case? Have you checked your configuration regarding completeness and validity? Can you see any exceptions in your logs?

      Best regards,
      Sven

Kommentar verfassen

Trage deine Daten unten ein oder klicke ein Icon um dich einzuloggen:

WordPress.com-Logo

Du kommentierst mit Deinem WordPress.com-Konto. Abmelden / Ändern )

Twitter-Bild

Du kommentierst mit Deinem Twitter-Konto. Abmelden / Ändern )

Facebook-Foto

Du kommentierst mit Deinem Facebook-Konto. Abmelden / Ändern )

Google+ Foto

Du kommentierst mit Deinem Google+-Konto. Abmelden / Ändern )

Verbinde mit %s