當前需要做個測試，mysql資料庫需要5000萬資料來完成效能測試，於是我們寫了個函式來完成這個功能。

函式定義如下：

#批量插入資料庫的函式
BEGIN
    #定義需要插入資料庫表中的欄位；
 
    DECLARE id BIGINT(20);
    DECLARE phone_number varchar(11);
 
    #定義變數控制插入資料庫表的行數
    DECLARE sid bigint(20) DEFAULT 1;
    DECLARE count bigint(20);
    declare total bigint(20) DEFAULT 50000000;

    #開啟迴圈判斷
    while sid < total do
        SET count = sid + 100000; //每次插入資料庫條數，防止條數太多提交失敗；

    #開始事務
    start transaction;
    WHILE (sid < count) DO //迴圈插入資料庫資料
    SET id = sid;
    SET phone_number = CONCAT(1, ROUND(ROUND(RAND(),10)*10000000000));
    INSERT INTO t_contact_phone(sid,phone_number)
    VALUES(id,phone_number);
    #累計行數
    SET sid = sid+1;
    END WHILE;
    commit;  //提交事務
  end while; //結束迴圈
END

執行上述函式即可建立5000萬條資料在測試庫用於測試。

當我們需要同時執行大批量操作，那如果依然順序單執行緒執行，會比較耗時，為了提高效率，我們需要通過多執行緒非同步方式來提高效率；

比方說大批量資料的同步方案，只能通過批量查詢+多執行緒+非同步的方式來完成。才能更短的時間內更高效的完成任務。

public class SynchronizationData {

private static final Logger logger = LoggerFactory.getLogger(SynchronizationData.class);
@Autowired
private ApplicationSumaryMapper applicationSumaryMapper
 
;//需要同步的資料表（mybatis完成）

@Autowired
private ContactPhoneMapper contactPhoneMapper;//需要插入的資料表一（mybatis完成）

@Autowired
private MiddleFieldMapper middleFieldMapper; //需要插入的資料表二（mybatis完成）

private static ExecutorService executor = Executors.newFixedThreadPool(2); //定義執行緒池用於多執行緒執行。

//需要同步總條數
private long totalNumber = 0;
//需要同步的最大進件id;
private long maxSid = 0;
//當前已經同步的sid
private long currentSid = 0;
private long currentPage = 0;
private long maxPage = 0;

public void startSynData() {

    //獲取當前系統中最大的sid
maxSid = applicationSumaryMapper.getMaxSid();
    logger.info("需要同步的最大進件id(SID)為:", maxSid);
    //根據sid獲取總條數
totalNumber = applicationSumaryMapper.getTotalNumber(maxSid);
    logger.info("需要同步的進件總數為:", totalNumber);
    maxPage = totalNumber / 1000 + 1;
    logger.info("最大頁數為:", maxPage);
    //判斷能否開始執行，執行時間為晚上12點到早上8點
boolean synStart = isSynStart(currentSid, maxSid);
    if (synStart) {
        //開始分頁查詢
for (long i = 0; i <= maxPage;) {
            List<ContactPhone> contactList = new ArrayList<ContactPhone>();
            List<MiddleField> middleList = new ArrayList<MiddleField>();
            currentPage = i;

            //查詢需要同步的資料集
            List<ApplicationSumary> synDataList = applicationSumaryMapper.getSynData(currentPage, currentSid, maxSid);
            //遍歷集合組裝資料
prepareDataForBatchInsert(contactList, middleList, synDataList);
            i = batchAddData(i, contactList, middleList, synDataList);
        }
    } else {
        //不符合同步條件，啥也不做
}
}

@Transactional //通過事務控制同成功，同失敗。
private long batchAddData(long i, List<ContactPhone> contactList, List<MiddleField> middleList, List<ApplicationSumary> synDataList) {
    //非同步批量插入contactList
Future<Integer> contactNumber = executor.submit(new contactPhoneCallable(contactList));
    //非同步批量插入middleList
Future<Integer> middleNumber = executor.submit(new middleFiledCallable(middleList));
    try {
        Integer number1 = contactNumber.get();
        Integer number2 = middleNumber.get();
        //如果都成功則繼續
if (number1 != -1 && number2 != -1) {
         i++;
        } else {
            //失敗則繼續當前頁執行
}

    } catch (Exception e) {
        e.printStackTrace();
        //異常並記錄當前的sid
logger.info("異常最小的sid為:"+synDataList.get(0).getSid());
        logger.info("異常最大的sid為:"+synDataList.get(synDataList.size()-1).getSid());
    }
    return i;
}
//遍歷要同步的資料集合組裝要批量插入的資料集合
private void prepareDataForBatchInsert(List<ContactPhone> contactList, List<MiddleField> middleList, List<ApplicationSumary> synDataList) {
    for (ApplicationSumary application : synDataList
            ) {
        MiddleField middleField = new MiddleField(application.getSid(), application.getUnitAddressCode(), application.getCustunitAddress(), application.getRegistAddressCode(), application.getRegistAddress()
                , application.getLiveAddressCode(), application.getLiveAddress(), application.getCustQq(), application.getCustWechatcode(), application.getAppStatus(), application.getAppStatusCode());
        middleList.add(middleField);

        //客戶電話號
if (application.getPhoneNubmer() != null) {
            ContactPhone phoneForSelf = new ContactPhone();
            contactList.add(phoneForSelf);
        };
        //銷售人員電話
if (application.getSalesPhone() != null) {
            ContactPhone phoneForSales = new ContactPhone();
            contactList.add(phoneForSales);
        };
        //家庭聯絡人電話
if (application.getFamilyPhone() != null) {
            ContactPhone phoneForFamily = new ContactPhone();
            contactList.add(phoneForFamily);
        };
        //緊急聯絡人電話
if (application.getOtherContactphone() != null) {
            ContactPhone phoneForOther = new ContactPhone();
            contactList.add(phoneForOther);
        };
        //工作證明人電話
if (application.getFalyPhone() != null) {
            ContactPhone phoneForFaly = new ContactPhone();
            contactList.add(phoneForFaly);
        };

    }
}
//判斷是否可以開始同步，這裡只允許晚上0點到早上八點允許執行
private boolean isSynStart(long currentSid, long maxSid) {
    Date date = new Date();
    int hours = date.getHours();
    if ((hours >= 0 || hours <= 7) && currentSid < maxSid) {
        return true;
    } else {
        return false;
    }
}

//多執行緒執行任務一：給聯絡人表中批量插入資料
class contactPhoneCallable implements Callable<Integer>{
     private List<ContactPhone> runList;

    public contactPhoneCallable(List<ContactPhone> runList) {
        this.runList = runList;
    }
    
    @Override
public Integer call() {
        try {
            int addNumber = contactPhoneMapper.batchAdd(runList);
            return addNumber;
        } catch (Exception e) {
            return -1;
        }
    }
}
//多執行緒執行的任務二；給中間資訊表中批量插入資料
class middleFiledCallable implements Callable<Integer>{
    private List<MiddleField> runList;

    public middleFiledCallable(List<MiddleField> runList) {
        this.runList = runList;
    }

    @Override
public Integer call() {
        try {
            int middleNumber = middleFieldMapper.batchAdd(runList);
            return middleNumber;
        } catch (Exception e) {
            return -1;
        }
    }
}

}

最後給出分頁查詢和批量插入的sql語句：

mybatis定義的方法：

List<ApplicationSumary> getSynData(@Param("currentPage") long currentPage, @Param("currentSid") long currentSid, @Param("maxSid") long maxSid);

分頁查詢sql:

<select id="getSynData" resultMap="BaseResultMap">
  select
  <include refid="Base_Column_List" />
  from t_application_sumary
  where sid<![CDATA[ <= ]]>#{maxSid} and sid> #{currentSid} limit #{currentPage},#{currentPage+1000}
</select>

mybatis定義的方法：

int batchAdd(List<ContactPhone> contactList);

批量插入sql:

<insert id="batchAdd" parameterType="java.util.List">
  insert into t_contact_phone (sid,phone_type,phone_number,phone_name)
  values
  <foreach collection="list" item="item" index="index" separator=",">
    (#{item.sid,jdbcType=BIGINT}, #{item.phoneType,jdbcType=VARCHAR}, #{item.phoneNumber,jdbcType=char},
    #{item.phoneName,jdbcType=VARCHAR})
  </foreach>
</insert>