除了binlog2sql工具外，使用python腳本閃回數據（數據庫誤操作）
阿新 • • 發佈：2018-03-21
eat 修改 ext pri 之前 測試 required find 實例
利用binlog日誌恢復數據庫誤操作數據
在人工手動進行一些數據庫寫操作的時候（比方說數據修改），尤其是一些不可控的批量更新或刪除，通常都建議備份後操作。不過不怕萬一，就怕一萬，有備無患總是好的。在線上或者測試環境誤操作導致數據被刪除或者更新後，想要恢復，一般有兩種方法。
方法一、利用最近的全量備份+增量binlog備份，恢復到誤操作之前的狀態，但是隨著數據量的增大，binlog的增多，恢復起來很費時。
方法二、如果binlog的格式為row，那麽就可以將binlog解析出來生成反向的原始SQL
以下是利用方法二寫的一個python腳本binlog_rollback.py，可利用此腳本生成反向的原始SQL。
說明：
0、前提是binlog的格式為row
1、要恢復的表操作前後表結構沒有發生變更，否則腳本無法解析
2、只生成DML(insert/update/delete)的rollback語句
3、最終生成的SQL是逆序的，所以最新的DML會生成在輸入文件的最前面，並且帶上了時間戳和偏移點，方便查找目標
4、需要提供一個連接MySQL的只讀用戶，主要是為了獲取表結構
5、如果binlog過大，建議帶上時間範圍，也可以指定只恢復某個庫的SQL
6、SQL生成後，請務必在測試環境上測試恢復後再應用到線上
數據庫配置：
[mysqld]
server_id = 1
log_bin = /var/lib/mysql/mysql-bin.log
max_binlog_size = 100M
binlog_format = row
binlog_row_image = full
腳本代碼：
創建py腳本文件binlog_rollback.py如下
#!/bin/env python # -*- coding:utf-8 -*- import os,sys,re,getopt import MySQLdb host = ‘127.0.0.1‘ user = ‘‘ password = ‘‘ port = 3306 start_datetime = ‘1971-01-01 00:00:00‘ stop_datetime = ‘2037-01-01 00:00:00‘ start_position = ‘4‘ stop_position = ‘18446744073709551615‘ database = ‘‘mysqlbinlog_bin = ‘mysqlbinlog -v‘ binlog = ‘‘ fileContent = ‘‘ output=‘rollback.sql‘ only_primary = 0 # ---------------------------------------------------------------------------------------- # 功能:獲取參數,生成相應的binlog解析文件 # ---------------------------------------------------------------------------------------- def getopts_parse_binlog(): global host global user global password global port global fileContent global output global binlog global start_datetime global stop_datetime global start_position global stop_position global database global only_primary try: options, args = getopt.getopt(sys.argv[1:], "f:o:h:u:p:P:d:", ["help","binlog=","output=","host=","user=","password=","port=","start-datetime=", "stop-datetime=","start-position=","stop-position=","database=","only-primary="]) except getopt.GetoptError: print "參數輸入有誤!!!!!" options = [] if options == [] or options[0][0] in ("--help"): usage() sys.exit() print "正在獲取參數....." for name, value in options: if name == "-f" or name == "--binlog": binlog = value if name == "-o" or name == "--output": output = value if name == "-h" or name == "--host": host = value if name == "-u" or name == "--user": user = value if name == "-p" or name == "--password": password = value if name == "-P" or name == "--port": port = value if name == "--start-datetime": start_datetime = value if name == "--stop-datetime": stop_datetime = value if name == "--start-position": start_position = value if name == "--stop-position": stop_position = value if name == "-d" or name == "--database": database = value if name == "--only-primary" : only_primary = value if binlog == ‘‘ : print "錯誤:請指定binlog文件名!" usage() if user == ‘‘ : print "錯誤:請指定用戶名!" usage() if password == ‘‘ : print "錯誤:請指定密碼!" usage() if database <> ‘‘ : condition_database = "--database=" + "‘" + database + "‘" else: condition_database = ‘‘ print "正在解析binlog....." fileContent=os.popen("%s %s --base64-output=DECODE-ROWS --start-datetime=‘%s‘ --stop-datetime=‘%s‘ --start-position=‘%s‘ --stop-position=‘%s‘ %s\ |grep ‘###‘ -B 2|sed -e ‘s/### //g‘ -e ‘s/^INSERT/##INSERT/g‘ -e ‘s/^UPDATE/##UPDATE/g‘ -e ‘s/^DELETE/##DELETE/g‘ " \ %(mysqlbinlog_bin,binlog,start_datetime,stop_datetime,start_position,stop_position,condition_database)).read() #print fileContent # ---------------------------------------------------------------------------------------- # 功能:初始化binlog裏的所有表名和列名,用全局字典result_dict來儲存每個表有哪些列 # ---------------------------------------------------------------------------------------- def init_col_name(): global result_dict global pri_dict global fileContent result_dict = {} pri_dict = {} table_list = re.findall(‘`.*`\\.`.*`‘,fileContent) table_list = list(set(table_list)) #table_list 為所有在這段binlog裏出現過的表 print "正在初始化列名....." for table in table_list: sname = table.split(‘.‘)[0].replace(‘`‘,‘‘) tname = table.split(‘.‘)[1].replace(‘`‘,‘‘) #連接數據庫獲取列和列id try: conn = MySQLdb.connect(host=host,user=user,passwd=password,port=int(port)) cursor = conn.cursor() cursor.execute("select ordinal_position,column_name \ from information_schema.columns where table_schema=‘%s‘ and table_name=‘%s‘ " %(sname,tname)) result=cursor.fetchall() if result == () : print ‘Warning:‘+sname+‘.‘+tname+‘已刪除‘ #sys.exit() result_dict[sname+‘.‘+tname]=result cursor.execute("select ordinal_position,column_name \ from information_schema.columns where table_schema=‘%s‘ and table_name=‘%s‘ and column_key=‘PRI‘ " %(sname,tname)) pri=cursor.fetchall() #print pri pri_dict[sname+‘.‘+tname]=pri cursor.close() conn.close() except MySQLdb.Error, e: try: print "Error %d:%s" % (e.args[0], e.args[1]) except IndexError: print "MySQL Error:%s" % str(e) sys.exit() #print result_dict #print pri_dict # ---------------------------------------------------------------------------------------- # 功能:拼湊回滾sql,逆序 # ---------------------------------------------------------------------------------------- def gen_rollback_sql(): global only_primary fileOutput = open(output, ‘w‘) #先將文件根據‘--‘分塊,每塊代表一個sql area_list=fileContent.split(‘--\n‘) #逆序讀取分塊 print "正在開始拼湊sql....." for area in area_list[::-1]: #由於一條sql可能影響多行,每個sql又可以分成多個逐條執行的sql sql_list = area.split(‘##‘) #先將pos點和timestamp傳入輸出文件中 for sql_head in sql_list[0].splitlines(): sql_head = ‘#‘+sql_head+‘\n‘ fileOutput.write(sql_head) #逐條sql進行替換更新,逆序 for sql in sql_list[::-1][0:-1]: try: if sql.split()[0] == ‘INSERT‘: rollback_sql = re.sub(‘^INSERT INTO‘, ‘DELETE FROM‘, sql, 1) rollback_sql = re.sub(‘SET\n‘, ‘WHERE\n‘, rollback_sql, 1) tablename_pos = 2 table_name = rollback_sql.split()[tablename_pos].replace(‘`‘, ‘‘) # 獲取該sql中的所有列 col_list = sorted(list(set(re.findall(‘@\d+‘, rollback_sql)))) # 因為第一個列前面沒有逗號或者and,所以單獨替換 rollback_sql = rollback_sql.replace(‘@1=‘, result_dict[table_name][0][1]+‘=‘) for col in col_list[1:]: i = int(col[1:]) - 1 rollback_sql = rollback_sql.replace(col+‘=‘, ‘AND ‘ + result_dict[table_name][i][1]+‘=‘,1) # 如果only_primary開啟且存在主鍵,where條件裏就只列出主鍵字段 if int(only_primary) == 1 and pri_dict[table_name] <> (): sub_where = ‘‘ for primary in pri_dict[table_name]: primary_name = primary[1] for condition in rollback_sql.split(‘WHERE‘, 1)[1].splitlines(): if re.compile(‘^\s*‘+primary_name).match(condition) or re.compile(‘^\s*AND\s*‘+primary_name).match(condition): sub_where = sub_where + condition + ‘\n‘ sub_where = re.sub(‘^\s*AND‘, ‘‘, sub_where, 1) rollback_sql = rollback_sql.split(‘WHERE‘, 1)[0] + ‘WHERE\n‘ + sub_where if sql.split()[0] == ‘UPDATE‘: rollback_sql = re.sub(‘SET\n‘, ‘#SET#\n‘, sql, 1) rollback_sql = re.sub(‘WHERE\n‘, ‘SET\n‘, rollback_sql, 1) rollback_sql = re.sub(‘#SET#\n‘, ‘WHERE\n‘, rollback_sql, 1) tablename_pos = 1 table_name = rollback_sql.split()[tablename_pos].replace(‘`‘, ‘‘) # 獲取該sql中的所有列 col_list = sorted(list(set(re.findall(‘@\d+‘, rollback_sql)))) # 因為第一個列前面沒有逗號或者and,所以單獨替換 rollback_sql = rollback_sql.replace(‘@1=‘, result_dict[table_name][0][1] + ‘=‘) for col in col_list[1:]: i = int(col[1:]) - 1 rollback_sql = rollback_sql.replace(col+‘=‘, ‘,‘ + result_dict[table_name][i][1]+‘=‘, 1).replace(col+‘=‘,‘AND ‘ +result_dict[table_name][i][1]+‘=‘) # 如果only_primary開啟且存在主鍵,where條件裏就只列出主鍵字段 if int(only_primary) == 1 and pri_dict[table_name] <> (): sub_where = ‘‘ for primary in pri_dict[table_name]: primary_name = primary[1] for condition in rollback_sql.split(‘WHERE‘, 1)[1].splitlines(): if re.compile(‘^\s*‘ + primary_name).match(condition) or re.compile(‘^\s*AND\s*‘+primary_name).match(condition): sub_where = sub_where + condition + ‘\n‘ sub_where = re.sub(‘^\s*AND‘, ‘‘, sub_where, 1) rollback_sql = rollback_sql.split(‘WHERE‘, 1)[0] + ‘WHERE\n‘ + sub_where if sql.split()[0] == ‘DELETE‘: rollback_sql = re.sub(‘^DELETE FROM‘, ‘INSERT INTO‘, sql, 1) rollback_sql = re.sub(‘WHERE\n‘, ‘SET\n‘, rollback_sql, 1) tablename_pos = 2 table_name = rollback_sql.split()[tablename_pos].replace(‘`‘, ‘‘) # 獲取該sql中的所有列 col_list = sorted(list(set(re.findall(‘@\d+‘, rollback_sql)))) # 因為第一個列前面沒有逗號或者and,所以單獨替換 rollback_sql = rollback_sql.replace(‘@1=‘, result_dict[table_name][0][1] + ‘=‘) for col in col_list[1:]: i = int(col[1:]) - 1 rollback_sql = rollback_sql.replace(col+‘=‘, ‘,‘ + result_dict[table_name][i][1]+‘=‘,1) rollback_sql = re.sub(‘\n$‘,‘;\n‘,rollback_sql) #print rollback_sql fileOutput.write(rollback_sql) except IndexError,e: print "Error:%s" % str(e) sys.exit() print "done!" def usage(): help_info="""========================================================================================== Command line options : --help # OUT : print help info -f, --binlog # IN : binlog file. (required) -o, --outfile # OUT : output rollback sql file. (default ‘rollback.sql‘) -h, --host # IN : host. (default ‘127.0.0.1‘) -u, --user # IN : user. (required) -p, --password # IN : password. (required) -P, --port # IN : port. (default 3306) --start-datetime # IN : start datetime. (default ‘1970-01-01 00:00:00‘) --stop-datetime # IN : stop datetime. default ‘2070-01-01 00:00:00‘ --start-position # IN : start position. (default ‘4‘) --stop-position # IN : stop position. (default ‘18446744073709551615‘) -d, --database # IN : List entries for just this database (No default value). --only-primary # IN : Only list primary key in where condition (default 0) Sample : shell> python binlog_rollback.py -f ‘mysql-bin.000001‘ -o ‘/tmp/rollback.sql‘ -h 192.168.0.1 -u ‘user‘ -p ‘pwd‘ -P 3307 -d dbname ==========================================================================================""" print help_info sys.exit() if __name__ == ‘__main__‘: getopts_parse_binlog() init_col_name() gen_rollback_sql()
實例：
mysql> grant select on *.* to ‘jeck‘@‘localhost‘ identified by ‘123‘; Query OK, 0 rows affected (0.09 sec)
mysql> create database t2; Query OK, 1 row affected (0.02 sec) mysql> use t2; Database changed mysql> create table test( -> id int(10) not null auto_increment, -> gender enum(‘male‘,‘woman‘), -> hobby varchar(20) default null, -> primary key(id) -> ); Query OK, 0 rows affected (0.64 sec) mysql> desc test; +--------+----------------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +--------+----------------------+------+-----+---------+----------------+ | id | int(10) | NO | PRI | NULL | auto_increment | | gender | enum(‘male‘,‘woman‘) | YES | | NULL | | | hobby | varchar(20) | YES | | NULL | | +--------+----------------------+------+-----+---------+----------------+ 3 rows in set (0.03 sec) mysql> alter table test add name varchar(15) after id; Query OK, 0 rows affected (0.58 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql> insert into test values -> (1,‘tom‘,‘male‘,‘movie‘), -> (2,‘Nancy‘,‘woman‘,‘dance‘), -> (3,‘jeck‘,‘male‘,‘basketball‘), -> (4,‘danny‘,‘male‘,‘game‘); Query OK, 4 rows affected (0.04 sec) Records: 4 Duplicates: 0 Warnings: 0 mysql> select * from test; +----+-------+--------+------------+ | id | name | gender | hobby | +----+-------+--------+------------+ | 1 | tom | male | movie | | 2 | Nancy | woman | dance | | 3 | jeck | male | basketball | | 4 | danny | male | game | +----+-------+--------+------------+ 4 rows in set (0.00 sec)
更新數據
mysql> update test set hobby=‘hike‘ where name=‘jeck‘; Query OK, 1 row affected (0.06 sec) Rows matched: 1 Changed: 1 Warnings: 0
刪除數據
mysql> delete from test where id=4; Query OK, 1 row affected (0.05 sec) mysql> select * from test; +----+-------+--------+-------+ | id | name | gender | hobby | +----+-------+--------+-------+ | 1 | tom | male | movie | | 2 | Nancy | woman | dance | | 3 | jeck | male | hike | +----+-------+--------+-------+ 3 rows in set (0.00 sec)
生成方向sql語句文件
[root@A mysql]# python binlog_rollback.py -f /var/lib/mysql/mysql-bin.000003 -o rollback.sql -u jeck -p 123 --start-datetime=‘2018-03-21 16:37:00‘ --stop-datetime=‘2018-03-21 16:39:00‘ -d t2 正在獲取參數..... 正在解析binlog..... 正在初始化列名..... 正在開始拼湊sql..... done!
數據恢復輸出文件 vim rollback.sql
## at 3553 ##180321 16:38:58 server id 1 end_log_pos 3605 CRC32 0xd8f72cdb Delete_rows: table id 72 flags: STMT_END_F INSERT INTO `t2`.`test` SET id=4 ,name=‘danny‘ ,gender=1 ,hobby=‘game‘; ## at 3324 ##180321 16:37:58 server id 1 end_log_pos 3398 CRC32 0xafd8a964 Update_rows: table id 72 flags: STMT_END_F UPDATE `t2`.`test` SET id=3 ,name=‘jeck‘ ,gender=1 ,hobby=‘basketball‘ WHERE id=3 AND name=‘jeck‘ AND gender=1 AND hobby=‘hike‘;
檢查語句並恢復
[root@A mysql]# mysql -uroot -p123 -S /var/lib/mysql/mysql.sock < rollback.sql
mysql> select * from t2.test; +----+-------+--------+------------+ | id | name | gender | hobby | +----+-------+--------+------------+ | 1 | tom | male | movie | | 2 | Nancy | woman | dance | | 3 | jeck | male | basketball | | 4 | danny | male | game | +----+-------+--------+------------+ 4 rows in set (0.00 sec)
轉載自：http://www.cnblogs.com/prayer21/p/6018736.html
除了binlog2sql工具外，使用python腳本閃回數據（數據庫誤操作）