首页 \ 问答 \ Java OutputStream文件使用httpclient上传性能(Java OutputStream file upload performances using httpclient)

Java OutputStream文件使用httpclient上传性能(Java OutputStream file upload performances using httpclient)

 我有一个在我的linux服务器（debian 7）的后台运行的java jar应用程序。 服务器通常通过apache接收文件，然后jar定期拉出数据库，使用来自httpclient（v2.5）的http连接池将文件上传到最终目的地。  
 服务器是16GB RAM，8核cpu，1To磁盘和2Gb / s互联网带宽。 我的问题是，jar的上传只占10％或20％的带宽。 经过多次调查后，我认为这是因为远程服务器的容量可能是瓶颈。 所以我想在我的服务器上启动更多线程以同时处理更多文件并使用我拥有的所有带宽，不幸的是，使用httpclient进行文件上传似乎正在吃掉很多cpu！  
 实际上，jar正在使用20个同时上传的可运行线程，并且cpu始终处于100％，如果我尝试启动更多线程，则平均负载增加并打破记录，使系统变得如此缓慢且无法使用。 奇怪的是，iowaits似乎是null，所以我真的不知道是什么导致了平均负载。  
 我只使用一个线程运行一个hprof，结果如下：  
 CPU SAMPLES BEGIN (total = 4617) Thu Jul 28 17:42:35 2016 rank self accum count trace method 1 52.76% 52.76% 2436 301157 java.net.SocketOutputStream.socketWrite0 2 33.53% 86.29% 1548 300806 java.net.SocketInputStream.socketRead0 3 1.62% 87.91% 75 301138 org.sqlite.core.NativeDB.step 4 1.47% 89.39% 68 301158 java.io.FileInputStream.readBytes 5 1.06% 90.45% 49 300078 java.lang.ClassLoader.defineClass1 6 0.26% 90.71% 12 300781 com.mysql.jdbc.SingleByteCharsetConverter.<clinit> 7 0.19% 90.90% 9 300386 java.lang.Throwable.fillInStackTrace 8 0.19% 91.10% 9 300653 java.lang.ClassLoader.loadClass 9 0.19% 91.29% 9 300780 com.mysql.jdbc.SingleByteCharsetConverter.<clinit> 10 0.17% 91.47% 8 300387 java.net.URLClassLoader.findClass 11 0.17% 91.64% 8 300389 java.util.zip.Inflater.inflateBytes 12 0.15% 91.79% 7 300090 java.lang.ClassLoader.findBootstrapClass 13 0.15% 91.94% 7 300390 java.util.zip.ZipFile.getEntry 14 0.13% 92.07% 6 300805 java.net.PlainSocketImpl.socketConnect  
 这些文件使用一个常见的httpclient POST执行请求发送，该请求使用来自filebody类的覆盖writeTo（）方法，该类使用8kb的bufferedInputStream。  
 您是否认为可以降低文件上传的性能影响，并解决我未使用的带宽问题？  
 在此先感谢您的帮助。 

I have a java jar application running in the background of my linux servers (debian 7). The servers receives files through apache generally, then the jar pull the database regularly to upload the files to their final destination using a pool of http connections from an httpclient (v2.5). 
The servers are 16gb of ram, 8 cores cpu, 1To disk and 2Gb/s internet bandwidth. My problem is that the bandwidth is only used at 10% or 20% for the uploads made by the jar. After many investigations I think it is because of the capacity of the distant server which might be the bottleneck. So I wanted to launch more threads on my servers to proceed more files at the same time and use all the bandwidth I have, unfortunately file upload with httpclient is eating a lot of cpu it seems !  
Actually the jars are working with 20 simultaneous upload runnable threads and the cpu is constantly at 100%, if I try to start more threads the load average increase and break records, getting the system so slow and unusable. Strange thing is the iowaits seems to be null, so I really don't know what is causing the load average. 
I have run an hprof using only one thread, here is the result :  
CPU SAMPLES BEGIN (total = 4617) Thu Jul 28 17:42:35 2016 rank self accum count trace method 1 52.76% 52.76% 2436 301157 java.net.SocketOutputStream.socketWrite0 2 33.53% 86.29% 1548 300806 java.net.SocketInputStream.socketRead0 3 1.62% 87.91% 75 301138 org.sqlite.core.NativeDB.step 4 1.47% 89.39% 68 301158 java.io.FileInputStream.readBytes 5 1.06% 90.45% 49 300078 java.lang.ClassLoader.defineClass1 6 0.26% 90.71% 12 300781 com.mysql.jdbc.SingleByteCharsetConverter.<clinit> 7 0.19% 90.90% 9 300386 java.lang.Throwable.fillInStackTrace 8 0.19% 91.10% 9 300653 java.lang.ClassLoader.loadClass 9 0.19% 91.29% 9 300780 com.mysql.jdbc.SingleByteCharsetConverter.<clinit> 10 0.17% 91.47% 8 300387 java.net.URLClassLoader.findClass 11 0.17% 91.64% 8 300389 java.util.zip.Inflater.inflateBytes 12 0.15% 91.79% 7 300090 java.lang.ClassLoader.findBootstrapClass 13 0.15% 91.94% 7 300390 java.util.zip.ZipFile.getEntry 14 0.13% 92.07% 6 300805 java.net.PlainSocketImpl.socketConnect  
the files are sent with a common httpclient POST execute request with an overrided writeTo() method from the filebody class that uses a bufferedInputStream of 8kb. 
Do you think it is possible to reduce the performance impact of the file uploads, and solve my problem of unused bandwidth ? 
Thanks in advance for your help.

原文：https://stackoverflow.com/questions/38641955

更新时间：2022-04-26 11:04

最满意答案

 你需要分解每个单词并进行SOUNDEX比较，这正是我要告诉你的这个功能所做的。  
 使用该功能  
 用法示例： SELECT p.name FROM products p WHERE soundex_match('fiftythree', p.name, ' ')  
 它需要3个参数：  
 
  针：你正在寻找的词  
  haysack：你正在搜索的字串  
  splitChar：将字符串拆分为单个单词的空白字符。 通常它是空间（''）  
 
 如果干草堆中的任何单词听起来与针类似，则函数将返回1，否则返回0。  
 在数据库中创建函数  
 因此，进入你的数据库（phpMyAdmin或命令行）并执行此操作，你只需要这样做一次）：  
drop function if exists soundex_match;
delimiter $$
create function soundex_match (needle varchar(128), haystack text, splitChar varchar(1)) returns tinyint
  deterministic
  begin
    declare spacePos int;
    declare searchLen int default length(haystack);
    declare curWord varchar(128) default '';
    declare tempStr text default haystack;
    declare tmp text default '';
    declare soundx1 varchar(64) default soundex(needle);
    declare soundx2 varchar(64) default '';

    set spacePos = locate(splitChar, tempStr);

    while searchLen > 0 do
      if spacePos = 0 then
        set tmp = tempStr;
        select soundex(tmp) into soundx2;
        if soundx1 = soundx2 then
          return 1;
        else
          return 0;
        end if;
      end if;

      if spacePos != 0 then
        set tmp = substr(tempStr, 1, spacePos-1);
        set soundx2 = soundex(tmp);
        if soundx1 = soundx2 then
          return 1;
        end if;
        set tempStr = substr(tempStr, spacePos+1);
        set searchLen = length(tempStr);
      end if;

      set spacePos = locate(splitChar, tempStr);

    end while;

    return 0;

  end
$$
delimiter ;
 
 http://www.imranulhoque.com/mysql/mysql-function-soundex-match-multi-word-string/ 

You'll need to break up each word and do the SOUNDEX comparison, which is exactly what this function I'm going to tell you about does.  
Using the function 
Example usage: SELECT p.name FROM products p WHERE soundex_match('fiftythree', p.name, ' ') 
It takes 3 arguments: 
 
 needle: The word you are looking for 
 haysack: The string of words among which you are searching 
 splitChar: The whitespace charater that’ll split the string into single words. Generally it is the space(‘ ‘) 
 
If any word in haystack sounds similar to needle, the function will return 1 and 0 otherwise. 
Creating the function in your database 
So go into your database (phpMyAdmin or the command line) and execute this, you only need to do it this one time): 
drop function if exists soundex_match;
delimiter $$
create function soundex_match (needle varchar(128), haystack text, splitChar varchar(1)) returns tinyint
  deterministic
  begin
    declare spacePos int;
    declare searchLen int default length(haystack);
    declare curWord varchar(128) default '';
    declare tempStr text default haystack;
    declare tmp text default '';
    declare soundx1 varchar(64) default soundex(needle);
    declare soundx2 varchar(64) default '';

    set spacePos = locate(splitChar, tempStr);

    while searchLen > 0 do
      if spacePos = 0 then
        set tmp = tempStr;
        select soundex(tmp) into soundx2;
        if soundx1 = soundx2 then
          return 1;
        else
          return 0;
        end if;
      end if;

      if spacePos != 0 then
        set tmp = substr(tempStr, 1, spacePos-1);
        set soundx2 = soundex(tmp);
        if soundx1 = soundx2 then
          return 1;
        end if;
        set tempStr = substr(tempStr, spacePos+1);
        set searchLen = length(tempStr);
      end if;

      set spacePos = locate(splitChar, tempStr);

    end while;

    return 0;

  end
$$
delimiter ;
 
http://www.imranulhoque.com/mysql/mysql-function-soundex-match-multi-word-string/

Java OutputStream文件使用httpclient上传性能(Java OutputStream file upload performances using httpclient)

最满意答案

使用该功能

在数据库中创建函数

Using the function

Creating the function in your database

相关问答

与逗号分隔的子查询的Soundex比较？(Soundex comparison with comma separated subquery?)[2022-04-19]

有什么数据库专业书籍介绍？[2022-05-19]

如何在Jooq中使用soundex函数？(How to use soundex function in Jooq?)[2022-01-22]

Symfony2注册的MySQL soundex失败了吗？(Symfony2 registered MySQL soundex fails?)[2023-09-29]

如何在MySQL中搜索Soundex（）子串？(How to search for Soundex() substrings in MySQL?)[2021-05-14]

soundex可以在mysql的列中使用吗？(can soundex be used on portion of a column in mysql?)[2023-11-19]

如何在mysql中查询soundex（）(How to query soundex() in mysql)[2024-03-08]

T-SQL Soundex / Difference查找重复行(T-SQL Soundex/Difference to find duplicate rows)[2021-12-18]

用相同的soundex替换单词(Replacing word with same soundex)[2021-12-16]

性能：将soundex放入DB或在查询后翻译？(Performance: Put soundex in DB or translate it after the query?)[2022-01-05]

相关文章

最新问答