1. 程式人生 > >TiDB數據庫 mydumper與loader導入數據

TiDB數據庫 mydumper與loader導入數據

mman 除了 copy other iat 多個 char over 端口被占用

mysql導出數據最好的方法是使用tidb官方的工具mydumper

導入tidb最好的方法是使用loader工具,大概19.4G每小時的速度。

詳細的步驟可以參考官網:https://pingcap.com/docs-cn/op-guide/migration-overview/

自己的總結:以前將mysq導出數據的時候,是一個大的sql文件,比如這次就是從mysql中導出一個41G大的文件,這是單個的。以為這樣子就能使用loader命令導入到tidb中,最後發現錯了。loader命令的導入是將一個目錄下的sql文件導入到數據庫中,那麽問題就來了,像我們之前那樣只有一個大的sql文件是不能使用loader命令導入數據庫的。(這裏的loader是tidb官方自己研發的)最後使用了source命令導入了這個大的sql文件。

所以說要使用loader命令導入數據,那麽前提是就要用mydumper命令導出數據,這樣的數據才能被loader導入進去。

mydumper命令是將一個表導出到一個目錄下面,這個表比較大的時候,mydumper就會分成以64M為大小的一個個的小文件(比如這次的41G的文件分成了600多個小文件)。

當然使用導入導出命令也踩了一些坑

報錯:

1、導入數據出現10084錯誤

技術分享圖片

這是因為端口被占用,導致不能重新寫入,使用netstat -lnp| grep 10084命令找到該端口10084的對應的進程,然後使用kill命令殺死進程即可。

2、tidb_loader數據庫下check_point表的作用

解釋:當我們導入tidb數據庫數據之後,這個check_point表都會產生一行數據,用來記錄導入數據的狀態信息以及導入時間等。

如果你導入一次數據之後,check_point以及記錄一次了,這個時候你把導入的那個表刪除了,但是沒有刪除check_point的導入記錄,那麽現在你繼續重新導入之前的那個表,會發現導入失敗。

下面是講解check_point表的作用:

mysql> select * from haha
    -> ;
+------+------+
| id   | name |
+------+------+
|    1 | chen |
|    2
| chao | | 3 | feng | +------+------+ 3 rows in set (0.00 sec) mysql> drop table haha; Query OK, 0 rows affected (0.25 sec) mysql> show databases; +--------------------+ | Database | +--------------------+ | INFORMATION_SCHEMA | | PERFORMANCE_SCHEMA | | mob | | mysql | | test | | tidb_loader | +--------------------+ 6 rows in set (0.00 sec) mysql> use tidb_loader; Database changed mysql> show tables; +-----------------------+ | Tables_in_tidb_loader | +-----------------------+ | checkpoint | +-----------------------+ 1 row in set (0.00 sec) mysql> select * from checkpoint; +--------+---------------+-----------+----------+--------+---------+---------------------+---------------------+ | id | filename | cp_schema | cp_table | offset | end_pos | create_time | update_time | +--------+---------------+-----------+----------+--------+---------+---------------------+---------------------+ | 32c716 | test.haha.sql | test | haha | 98 | 98 | 2018-11-16 12:00:08 | 2018-11-16 12:00:08 | +--------+---------------+-----------+----------+--------+---------+---------------------+---------------------+ 1 row in set (0.00 sec) mysql> exit Bye [tidb@:vg_adn_tidbCkhsTest:54.158.254.36:172.31.30.62 /usr/local/tidb-tools]$sudo bin/loader -h 127.0.0.1 -u root -P 4000 -t 1 -d /data/haha 2018/11/16 12:01:20 printer.go:52: [info] Welcome to loader 2018/11/16 12:01:20 printer.go:53: [info] Release Version: v1.0.0-47-gbaf98f6 2018/11/16 12:01:20 printer.go:54: [info] Git Commit Hash: baf98f63ae77b2d9a22e00bb3d1555452b21e4f9 2018/11/16 12:01:20 printer.go:55: [info] Git Branch: master 2018/11/16 12:01:20 printer.go:56: [info] UTC Build Time: 2018-11-12 01:34:40 2018/11/16 12:01:20 printer.go:57: [info] Go Version: go version go1.11 linux/amd64 2018/11/16 12:01:20 main.go:51: [info] config: {"log-level":"info","log-file":"","pprof-addr":":10084","pool-size":1,"dir":"/data/haha","db":{"host":"127.0.0.1","user":"root","port":4000},"checkpoint-schema":"tidb_loader","config-file":"","alternative-db":"","source-db":"","route-rules":[],"do-table":null,"do-db":null,"ignore-table":null,"ignore-db":null,"rm-checkpoint":false} 2018/11/16 12:01:20 checkpoint.go:82: [info] [loader] load checkpoint takes 0.005710 seconds 2018/11/16 12:01:20 loader.go:524: [info] [loader] prepare takes 0.000085 seconds 2018/11/16 12:01:20 checkpoint.go:207: [info] calc checkpoint finished. finished tables (map[test.haha:{}]) 2018/11/16 12:01:20 loader.go:685: [info] [loader][run db schema]/data/haha/test-schema-create.sql[start] 2018/11/16 12:01:20 db.go:112: [warning] [exec][sql][CREATE DATABASE `test` /* !40100 DEFAULT CHARACTER SET utf8 */;][error]Error 1007: Cant create database test; database exists 2018/11/16 12:01:20 loader.go:689: [info] [loader][database already exists, skip]/data/haha/test-schema-create.sql 2018/11/16 12:01:20 loader.go:694: [info] [loader][run db schema]/data/haha/test-schema-create.sql[finished] 2018/11/16 12:01:20 loader.go:704: [info] table (test.haha) has finished, skip. 2018/11/16 12:01:20 loader.go:750: [info] [loader] create tables takes 0.004609 seconds 2018/11/16 12:01:20 loader.go:757: [info] [loader] all data files have been dispatched, waiting for them finished 2018/11/16 12:01:20 loader.go:760: [info] [loader] all data files has been finished, takes 0.004650 seconds 2018/11/16 12:01:20 main.go:88: [info] loader stopped and exits [tidb@:vg_adn_tidbCkhsTest:54.158.254.36:172.31.30.62 /usr/local/tidb-tools]$mysql -u root -P 4000 -h 127.0.0.1 Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2564 Server version: 5.7.10-TiDB-v2.0.8 MySQL Community Server (Apache License 2.0) Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type help; or \h for help. Type \c to clear the current input statement. mysql> use test Database changed mysql> show tables; Empty set (0.00 sec) mysql>

TiDB數據庫 mydumper與loader導入數據