我的JBoss服务器在Linux上遇到100％的SYS CPU;(My JBoss server hits 100% SYS CPU on Linux; what can cause this?)

 我们一直在调试这个JBoss服务器问题。 经过大约10个小时的工作，服务器进入100％CPU恐慌攻击并停止。 在此期间，您无法运行任何新程序，因此您甚至无法使用kill -quit来获取堆栈跟踪。 这些高100％SYS CPU负载持续10-20秒，每隔几分钟重复一次。 
 
 我们已经工作了很长一段时间。 我们怀疑它与GC有关，但无法通过较小的程序确认。 我们使用-client和ParNew GC在i386 32位，RHEL5和Java 1.5.0_10上运行。 
 
 以下是我们迄今为止所尝试的内容： 
 
 
 
 我们限制了CPU亲和力，因此我们可以在高负载点击时实际使用服务器。 随着strace我们看到一个无限循环的SIGSEGV然后sig返回。 
 
 
 我们尝试使用Java程序重现这一点。 确实，SYS CPU％使用WeakHashMap或访问空指针时爬升得很高。 问题是fillStackTrace占用了大量用户CPU％，这就是我们从未达到过100％SYS CPU的原因。 
 
 
 我们知道，经过10个小时的压力，GC会变得疯狂，完整的GC有时需要5秒钟。 所以我们假设它与记忆有关。 
 
 
 在此期间的jstack显示所有线程都被阻止。 pstack在那段时间里偶尔出现MarkSweep堆栈跟踪，所以我们也不能确定这一点。 发送SIGQUIT没有任何结果：Java在SYS％加载周期结束后转储了堆栈跟踪。 
 
 
 我们现在尝试用一小段代码重现这个问题，这样我们就可以问Sun. 
 
 如果您知道是什么原因造成的，请告诉我们。 我们对想法持开放态度，我们无能为力，欢迎任何想法:) 
 
 谢谢你的时间。 

We've been debugging this JBoss server problem for quite a while. After about 10 hours of work, the server goes into 100% CPU panic attacks and just stalls. During this time you cannot run any new programs, so you can't even kill -quit to get a stack trace. These high 100% SYS CPU loads last 10-20 seconds and repeat every few minutes.
 
We have been working on for quite a while. We suspect it has something to do with the GC, but cannot confirm it with a smaller program. We are running on i386 32bit, RHEL5 and Java 1.5.0_10 using -client and ParNew GC.
 
Here's what we have tried so far:
 
 
 
We limited the CPU affinity so we can actually use the server when the high load hits. With strace we see an endless loop of SIGSEGV and then the sig return. 
 
 
We tried to reproduce this with a Java program. It's true that SYS CPU% climbs high with WeakHashMap or when accessing null pointers. Problem was that fillStackTrace took a lot of user CPU% and that's why we never reached 100% SYS CPU.
 
 
We know that after 10 hours of stress, GC goes crazy and full GC sometimes takes 5 seconds. So we assume it has something to do with memory.
 
 
jstack during that period showed all threads as blocked. pstack during that time, showed MarkSweep stack trace occasionally, so we can't be sure about this as well. Sending SIGQUIT yielded nothing: Java dumped the stack trace AFTER the SYS% load period was over.
 
 
We're now trying to reproduce this problem with a small fragment of code so we can ask Sun. 
 
If you know what's causing it, please let us know. We're open to ideas and we are clueless, any idea is welcome :)
 
Thanks for your time. 

原文：https://stackoverflow.com/questions/654872