TomEE对太多的@Asynchronous操作感到窒息(TomEE chokes on too many @Asynchronous operations)

 我正在使用Apache TomEE 1.5.2 JAX-RS，几乎是开箱即用的，具有预定义的HSQLDB。 
 
 以下是简化代码。 我有一个REST风格的接口，用于接收信号： 
 
@Stateless
@Path("signal")
public class SignalEndpoint {
    @Inject
    private SignalStore store;

    @POST
    public void post() {
        store.createSignal();
    }
}
 
 接收信号会触发很多东西。 商店将创建一个实体，然后触发异步事件。 
 
public class SignalStore {
    @PersistenceContext
    private EntityManager em;

    @EJB
    private EventDispatcher dispatcher;

    @Inject
    private Event<SignalEntity> created;

    public void createSignal() {
        SignalEntity entity = new SignalEntity();
        em.persist(entity);
        dispatcher.fire(created, entity);
    }
}
 
 调度程序非常简单，仅存在使事件处理异步。 
 
@Stateless
public class EventDispatcher {
    @Asynchronous
    public <T> void fire(Event<T> event, T parameter) {
        event.fire(parameter);
    }
}
 
 接收事件是另一回事，它从信号中获取数据，存储它，并触发另一个异步事件： 
 
@Stateless
public class DerivedDataCreator {
    @PersistenceContext
    private EntityManager em;

    @EJB
    private EventDispatcher dispatcher;

    @Inject
    private Event<DerivedDataEntity> created;

    @Asynchronous
    public void onSignalEntityCreated(@Observes SignalEntity signalEntity) {
        DerivedDataEntity entity = new DerivedDataEntity(signalEntity);
        em.persist(entity);
        dispatcher.fire(created, entity);
    }
}
 
 对此的反应甚至是实体创建的第三层。 
 
 总而言之，我有一个REST调用，它同步创建一个SignalEntity ，它异步触发DerivedDataEntity的创建， DerivedDataEntity异步触发创建第三种类型的实体。 它完美地运行，存储过程完美地分离。 
 
 除了我以编程方式触发for循环中的大量（fe 1000）信号时。 根据我的AsynchronousPool大小，在处理大约一半大小的信号（非常快）后，应用程序完全冻结了几分钟。 然后它重新开始，在再次冻结之前快速处理大约相同数量的信号。 
 
 过去半小时我一直在玩AsynchronousPool设置。 例如，将其设置为2000可以轻松地立即处理所有信号，而不会冻结。 但在那之后，系统也不健全。 触发另外1000个信号，导致它们被创建得很好，但是从未发生过派生数据的整个创建。 
 
 现在我完全不知道该怎么做。 我当然可以摆脱所有这些异步事件并自己实现某种队列，但我一直认为EE容器的意义在于减轻我的烦恼。 异步EJB事件应该已经带来了自己的队列机制。 一个队列太满时不应该冻结的一个。 
 
 有任何想法吗？ 
 
 更新： 
 
 我现在用1.6.0-SNAPSHOT尝试过它。 它的行为有点不同：它仍然不起作用，但我得到一个例外： 
 
Aug 01, 2013 3:12:31 PM org.apache.openejb.core.transaction.EjbTransactionUtil handleSystemException
SEVERE: EjbTransactionUtil.handleSystemException: fail to allocate internal resource to execute the target task
javax.ejb.EJBException: fail to allocate internal resource to execute the target task
    at org.apache.openejb.async.AsynchronousPool.invoke(AsynchronousPool.java:81)
    at org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:240)
    at org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:86)
    at org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:303)
    at <<... my code ...>>
    ...
Caused by: java.util.concurrent.RejectedExecutionException: Timeout waiting for executor slot: waited 30 seconds
    at org.apache.openejb.util.executor.OfferRejectedExecutionHandler.rejectedExecution(OfferRejectedExecutionHandler.java:55)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
    at org.apache.openejb.async.AsynchronousPool.invoke(AsynchronousPool.java:75)
    ... 38 more
 
 好像TomEE不会做任何排队操作。 如果在通话时没有线程可以自由处理，那么运气不好。 当然，这不是故意的吗？ 
 
 更新2： 
 
 好吧，我似乎偶然发现了一个半解决方案：将AsynchronousPool.QueueSize属性设置为maxint可以解决冻结问题。 但问题仍然存在：为什么QueueSize首先如此受限制，更令人担忧的是：为什么这会阻止整个应用程序？ 如果队列已满，它会阻塞，但是一旦从中获取任务，另一个应该弹出，对吧？ 队列似乎被阻塞，直到它再次完全为空。 
 
 更新3： 
 
 对于任何想要去的人： http ： //github.com/JanDoerrenhaus/tomeefreezetestcase 
 
 更新4： 
 
 事实证明，增加队列大小并不能解决问题，它只会延迟它。 问题仍然存在：同时进行的异步操作太多，而且TomEE非常糟糕，以至于它甚至无法在终止时取消部署应用程序。 
 
 到目前为止，我的诊断是任务清理无法正常工作。 我的任务都非常小而且快（参见github上的测试用例 ）。 我已经担心OpenJPA或HSQLDB会减慢太多的并发呼叫，但我注释掉了所有的em.persist呼叫，问题仍然存在。 因此，如果我的任务非常小而且速度很快，但仍然设法阻止TomEE如此糟糕以至于在30秒后它无法获得任何进一步的任务（ javax.ejb.EJBException: fail to allocate internal resource to execute the target task ），我想，完成的任务会徘徊，堵塞管道，可以这么说。 
 
 我该如何解决这个问题？ 

I am using Apache TomEE 1.5.2 JAX-RS, pretty much out of the box, with the predefined HSQLDB.
 
The following is simplified code. I have a REST-style interface for receiving signals:
 
@Stateless
@Path("signal")
public class SignalEndpoint {
    @Inject
    private SignalStore store;

    @POST
    public void post() {
        store.createSignal();
    }
}
 
Receiving a signal triggers a lot of stuff. The store will create an entity, then fire an asynchronous event.
 
public class SignalStore {
    @PersistenceContext
    private EntityManager em;

    @EJB
    private EventDispatcher dispatcher;

    @Inject
    private Event<SignalEntity> created;

    public void createSignal() {
        SignalEntity entity = new SignalEntity();
        em.persist(entity);
        dispatcher.fire(created, entity);
    }
}
 
The dispatcher is very simple, and merely exists to make the event handling asynchronous.
 
@Stateless
public class EventDispatcher {
    @Asynchronous
    public <T> void fire(Event<T> event, T parameter) {
        event.fire(parameter);
    }
}
 
Receiving the event is something else, which derives data from the signal, stores it, and fires another asynchronous event:
 
@Stateless
public class DerivedDataCreator {
    @PersistenceContext
    private EntityManager em;

    @EJB
    private EventDispatcher dispatcher;

    @Inject
    private Event<DerivedDataEntity> created;

    @Asynchronous
    public void onSignalEntityCreated(@Observes SignalEntity signalEntity) {
        DerivedDataEntity entity = new DerivedDataEntity(signalEntity);
        em.persist(entity);
        dispatcher.fire(created, entity);
    }
}
 
Reacting to that is even a third layer of entity creation.
 
To summarize, I have a REST call, which synchronously creates a SignalEntity, which asynchronously triggers the creation of a DerivedDataEntity, which asynchronously triggers the creation of a third type of entity. It all works perfectly, and the storage processes are beautifully decoupled.
 
Except for when I programmatically trigger a lot (f.e. 1000) of signals in a for-loop. Depending on my AsynchronousPool size, after processing signals (quite fast) in the amount of about half of that size, the application completely freezes for up to some minutes. Then it resumes, to process about the same amount of signals, quite fast, before freezing again.
 
I have been playing around with AsynchronousPool settings for the last half hour. Setting it to 2000, for instance, will easily make all my signals be processed at once, without any freezes. But the system isn't sane either, after that. Triggering another 1000 signals, resulted in them being created allright, but the entire creation of derived data never happened.
 
Now I am completely at a loss as to what to do. I can of course get rid of all those asynchronous events and implement some sort of queue myself, but I always thought the point of an EE container was to relieve me of such tedium. Asynchronous EJB events should already bring their own queue mechanism. One which should not freeze as soon as the queue is too full.
 
Any ideas?
 
UPDATE:
 
I have now tried it with 1.6.0-SNAPSHOT. It behaves a little bit differently: It still doesn't work, but I do get an exception:
 
Aug 01, 2013 3:12:31 PM org.apache.openejb.core.transaction.EjbTransactionUtil handleSystemException
SEVERE: EjbTransactionUtil.handleSystemException: fail to allocate internal resource to execute the target task
javax.ejb.EJBException: fail to allocate internal resource to execute the target task
    at org.apache.openejb.async.AsynchronousPool.invoke(AsynchronousPool.java:81)
    at org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:240)
    at org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:86)
    at org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:303)
    at <<... my code ...>>
    ...
Caused by: java.util.concurrent.RejectedExecutionException: Timeout waiting for executor slot: waited 30 seconds
    at org.apache.openejb.util.executor.OfferRejectedExecutionHandler.rejectedExecution(OfferRejectedExecutionHandler.java:55)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
    at org.apache.openejb.async.AsynchronousPool.invoke(AsynchronousPool.java:75)
    ... 38 more
 
It is as though TomEE would not do ANY queueing of operations. If no thread is free to process in the moment of the call, tough luck. Surely, this cannot be intended..?
 
UPDATE 2:
 
Okay, I seem to have stumbled upon a semi-solution: Setting the AsynchronousPool.QueueSize property to maxint solves the freeze. But questions remain: Why is the QueueSize so limited in the first place, and, more worryingly: Why would this block the entire application? If the queue is full, it blocks, but as soon as a task is taken from it, another should pop in, right? The queue appears to be blocked until it is completely empty again.
 
UPDATE 3:
 
For anyone who wants to have a go: http://github.com/JanDoerrenhaus/tomeefreezetestcase
 
UPDATE 4:
 
As it turns out, increasing the queue size does NOT solve the problem, it merely delays it. The problem remains the same: Too many asynchronous operations at once, and TomEE chockes so bad, that it cannot even undeploy the application on termination anymore.
 
So far, my diagnosis is that the task cleanup does not work properly. My tasks are all very small and fast (see the test case on github). I was already afraid that it would be OpenJPA or HSQLDB slowing down on too many concurrent calls, but I commented out all em.persist calls, and the problem remained the same. So if my tasks are quite small and fast, but still manage to block out TomEE so bad that it could not get any further task in after 30 seconds (javax.ejb.EJBException: fail to allocate internal resource to execute the target task), I would imagine that completed tasks linger, clogging up the pipe, so to speak.
 
How could I resolve this issue?

原文：https://stackoverflow.com/questions/17979259