1. 程式人生 > >火車頭採集器使用‘匯入資料庫’形式釋出資料到帝國CMS資料庫的過程(原創)適用於ECMS7.2

火車頭採集器使用‘匯入資料庫’形式釋出資料到帝國CMS資料庫的過程(原創)適用於ECMS7.2

網上有不少相關的資訊,有部分文章已經相當不錯了,但是還是並不能正確的寫入記錄,或者說只能相容比較老的版本(具體是之前哪個版本,還並不知曉)(本文描述的內容,是可以適用於EmpireCMS7.2的),比如有一種說法,只需要修改:phome_ecms_news(一次insert,一次update),phome_ecms_news_data_1(一次insert)兩張表,現實是如果只操作這兩張表,一般都會出現如下的錯誤:

而且在刪除資料的時候,一樣會報一些資料庫方面的錯誤。這種情況下唯一的辦法就是檢視清楚,每一條新錄入的資料,在錄入過程中,到底對資料庫操作了什麼?

開啟SQL語句日誌功能,可以檢視在執行新增一條新資訊的時候,資料庫到底做了哪些操作,

SET GLOBAL log_output = 'TABLE';SET GLOBAL general_log = 'ON';(記得之後要關閉,不然會產生太多不必要的日誌檔案)

執行完成後,發現新增記錄過程中,所有執行過的SQL語句如下:

mysql> SELECT * from mysql.general_log ORDER BY event_time DESC; +----------------------------+------------------------------+-----------+-----------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | event_time | user_host | thread_id | server_id | command_type | argument | +----------------------------+------------------------------+-----------+-----------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 2018-10-11 13:45:14.338256 | root[root] @ localhost [::1] | 1344 | 1 | Query | SET GLOBAL general_log = 'OFF' | | 2018-10-11 13:45:09.951519 | root[root] @ localhost [::1] | 1653 | 1 | Quit | | | 2018-10-11 13:45:09.949531 | root[root] @ localhost [::1] | 1653 | 1 | Query | select andauth from phome_enewsuserloginck where userid='1' | | 2018-10-11 13:45:09.948513 | root[root] @ localhost [::1] | 1653 | 1 | Query | select userid,groupid,classid,userprikey,uprnd from phome_enewsuser where userid='1' and username='admin' and rnd='og7Q9XlQgP4v7l4Aq1mQ' and checked=0 limit 1 | | 2018-10-11 13:45:09.947509 | root[root] @ localhost [::1] | 1653 | 1 | Init DB | empirecms | | 2018-10-11 13:45:09.946506 | root[root] @ localhost [::1] | 1653 | 1 | Query | SET character_set_connection=utf8,character_set_results=utf8,character_set_client=binary,sql_mode='' | | 2018-10-11 13:45:09.945504 | [root] @ localhost [::1] | 1653 | 1 | Connect |

[email protected] on using TCP/IP | | 2018-10-11 13:45:09.888351 | root[root] @ localhost [::1] | 1652 | 1 | Quit | | | 2018-10-11 13:45:09.886346 | root[root] @ localhost [::1] | 1652 | 1 | Query | select sitename from phome_enewsbefrom | | 2018-10-11 13:45:09.885342 | root[root] @ localhost [::1] | 1652 | 1 | Query | select writer from phome_enewswriter | | 2018-10-11 13:45:09.884339 | root[root] @ localhost [::1] | 1652 | 1 | Query | select firsttitlename,isgoodname from phome_enewspublic limit 1 | | 2018-10-11 13:45:09.883337 | root[root] @ localhost [::1] | 1652 | 1 | Query | select typeid,tname from phome_enewsinfotype where mid='1' order by myorder | | 2018-10-11 13:45:09.883337 | root[root] @ localhost [::1] | 1652 | 1 | Query | select tempid,tempname from phome_enewsnewstemp order by modid,tempid | | 2018-10-11 13:45:09.883337 | root[root] @ localhost [::1] | 1652 | 1 | Query | select tempid,tempname from phome_enewsvotetemp order by tempid | | 2018-10-11 13:45:09.882334 | root[root] @ localhost [::1] | 1652 | 1 | Query | select definfovoteid from phome_enewsclass where classid='2' | | 2018-10-11 13:45:09.882334 | root[root] @ localhost [::1] | 1652 | 1 | Query | select ttids from phome_enewsclassadd where classid='2' | | 2018-10-11 13:45:09.880329 | root[root] @ localhost [::1] | 1652 | 1 | Query | select groupid,adminclass from phome_enewsuser where userid='1' limit 1 | | 2018-10-11 13:45:09.880329 | root[root] @ localhost [::1] | 1652 | 1 | Query | select doall,doselfinfo,doaddinfo,doeditinfo,dodelinfo,docheckinfo,dogoodinfo,dodocinfo,domoveinfo,domustcheck,docheckedit from phome_enewsgroup where groupid='1' | | 2018-10-11 13:45:09.880329 | root[root] @ localhost [::1] | 1652 | 1 | Query | select groupid,groupname from phome_enewsmembergroup order by level | | 2018-10-11 13:45:09.879326 | root[root] @ localhost [::1] | 1652 | 1 | Query | select userid,groupid,classid,userprikey,uprnd from phome_enewsuser where userid='1' and username='admin' and rnd='og7Q9XlQgP4v7l4Aq1mQ' and checked=0 limit 1 | | 2018-10-11 13:45:09.879326 | root[root] @ localhost [::1] | 1652 | 1 | Query | select andauth from phome_enewsuserloginck where userid='1' | | 2018-10-11 13:45:09.878323 | root[root] @ localhost [::1] | 1652 | 1 | Query | SET character_set_connection=utf8,character_set_results=utf8,character_set_client=binary,sql_mode='' | | 2018-10-11 13:45:09.878323 | root[root] @ localhost [::1] | 1652 | 1 | Init DB | empirecms | | 2018-10-11 13:45:09.877320 | [root] @ localhost [::1] | 1652 | 1 | Connect |
[email protected]
on using TCP/IP | | 2018-10-11 13:45:07.798427 | root[root] @ localhost [::1] | 1651 | 1 | Quit | || 2018-10-11 13:45:07.796423 | root[root] @ localhost [::1] | 1651 | 1 | Query | insert into phome_enewsdolog(username,logip,logtime,enews,doing,pubid,ipport) values('admin','::1','2018-10-11 13:45:07','AddNews','classid=2<br>id=2<br>title=title','1000010000000002','63863') | | 2018-10-11 13:45:07.794417 | root[root] @ localhost [::1] | 1651 | 1 | Query | select id,classid,ttid,onclick,plnum,totaldown,newspath,filename,userid,username,firsttitle,isgood,ispic,istop,isqf,ismember,isurl,truetime,lastdotime,havehtml,groupid,userfen,titlefont,titleurl,stb,fstb,restb,keyboard,title,ftitle,newstime,titlepic,smalltext,diggtop from phome_ecms_news where isgood>0 and ispic=1 order by newstime desc limit 4 | | 2018-10-11 13:45:07.793414 | root[root] @ localhost [::1] | 1651 | 1 | Query | select tempid,modid,temptext,showdate,listvar,subnews,rownum,docode from phome_enewsbqtemp where tempid='12' | | 2018-10-11 13:45:07.793414 | root[root] @ localhost [::1] | 1651 | 1 | Query | select classid,classname,islast,sonclass,tbname,intro,classimg,infos from phome_enewsclass where bclassid='0' and showclass=0 order by myorder,classid | | 2018-10-11 13:45:07.790407 | root[root] @ localhost [::1] | 1651 | 1 | Query | select bq,funname from phome_enewsbq where isclose=0 order by bqid | | 2018-10-11 13:45:07.789403 | root[root] @ localhost [::1] | 1651 | 1 | Query | select temptext,showdate from phome_enewsnewstemp where tempid='1' limit 1 | | 2018-10-11 13:45:07.789403 | root[root] @ localhost [::1] | 1651 | 1 | Query | select myvar,varvalue from phome_enewstempvar where isclose=0 order by myorder desc,varid | | 2018-10-11 13:45:07.788401 | root[root] @ localhost [::1] | 1651 | 1 | Query | select ttid,onclick,plnum,totaldown,newspath,filename,userid,username,firsttitle,isgood,ispic,istop,isqf,ismember,isurl,truetime,lastdotime,havehtml,groupid,userfen,titlefont,titleurl,stb,fstb,restb,keyboard,title,titlepic,newstime,ftitle,smalltext,diggtop from phome_ecms_news where id='2' limit 1 | | 2018-10-11 13:45:07.788401 | root[root] @ localhost [::1] | 1651 | 1 | Query | select keyid,dokey,newstempid,closepl,infotags,writer,befrom,newstext from phome_ecms_news_data_1 where id='2' limit 1 || 2018-10-11 13:45:07.787398 | root[root] @ localhost [::1] | 1651 | 1 | Query | update phome_enewspublic_update set lastnuminfo=lastnuminfo+1,lastnuminfotb='|1,38|',todaynuminfo=todaynuminfo+1 limit 1 || 2018-10-11 13:45:07.787398 | root[root] @ localhost [::1] | 1651 | 1 | Query | update phome_enewsfile_1 set pubid='1000010000000002',classid='2',id='2',cjid=0 where cjid='1539236638' || 2018-10-11 13:45:07.787398 | root[root] @ localhost [::1] | 1651 | 1 | Query | update phome_ecms_news set filename='2',titleurl='/youhua/seo/2018-10-11/2.html' where id='2' | | 2018-10-11 13:45:07.787398 | root[root] @ localhost [::1] | 1651 | 1 | Query | select count(*) as total from phome_enewsinfovote where pubid='1000010000000002' limit 1 || 2018-10-11 13:45:07.786395 | root[root] @ localhost [::1] | 1651 | 1 | Query | update phome_enewsclass set allinfos=allinfos+1,infos=infos+1 where classid='2' limit 1 | | 2018-10-11 13:45:07.786395 | root[root] @ localhost [::1] | 1651 | 1 | Query | select lasttimeinfo,lastnuminfotb,todaytimeinfo,todaytimepl,todaynuminfo,todaynumpl from phome_enewspublic_update limit 1 || 2018-10-11 13:45:07.785392 | root[root] @ localhost [::1] | 1651 | 1 | Query | insert into phome_ecms_news(id,classid,ttid,onclick,plnum,totaldown,newspath,filename,userid,username,firsttitle,isgood,ispic,istop,isqf,ismember,isurl,truetime,lastdotime,havehtml,groupid,userfen,titlefont,titleurl,stb,fstb,restb,keyboard,title,ftitle,newstime,titlepic,smalltext) values('2','2','0','0',0,'0','2018-10-11','','1','admin','0','0','0','0','0',0,'0','1539236707','1539236707','1','0','0','','','1','1','1','keyword','title','fubiao','1539236638','','content desc') || 2018-10-11 13:45:07.785392 | root[root] @ localhost [::1] | 1651 | 1 | Query | insert into phome_ecms_news_data_1(id,classid,keyid,dokey,newstempid,closepl,haveaddfen,infotags,writer,befrom,newstext) values('2','2','','1','0','0',0,'','','','<p>&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content</p>') || 2018-10-11 13:45:07.784391 | root[root] @ localhost [::1] | 1651 | 1 | Query | insert into phome_ecms_news_index(classid,checked,newstime,truetime,lastdotime,havehtml) values('2','1','1539236638','1539236707','1539236707','1') | | 2018-10-11 13:45:07.783387 | root[root] @ localhost [::1] | 1651 | 1 | Query | select sonclass from phome_enewsmod where mid='1' | | 2018-10-11 13:45:07.783387 | root[root] @ localhost [::1] | 1651 | 1 | Query | select id from phome_ecms_news where (title like '%keyword%') and (classid in (2)) and id<>0 order by newstime desc limit 10 | | 2018-10-11 13:45:07.781381 | root[root] @ localhost [::1] | 1651 | 1 | Query | select groupid,adminclass from phome_enewsuser where userid='1' limit 1 | | 2018-10-11 13:45:07.781381 | root[root] @ localhost [::1] | 1651 | 1 | Query | select doall,doselfinfo,doaddinfo,doeditinfo,dodelinfo,docheckinfo,dogoodinfo,dodocinfo,domoveinfo,domustcheck,docheckedit from phome_enewsgroup where groupid='1' | | 2018-10-11 13:45:07.781381 | root[root] @ localhost [::1] | 1651 | 1 | Query | select classid,modid,listdt,haddlist,sametitle,addreinfo,wburl,repreinfo from phome_enewsclass where classid='2' and islast=1 limit 1 | | 2018-10-11 13:45:07.773361 | root[root] @ localhost [::1] | 1651 | 1 | Query | select userid,groupid,classid,userprikey,uprnd from phome_enewsuser where userid='1' and username='admin' and rnd='og7Q9XlQgP4v7l4Aq1mQ' and checked=0 limit 1 | | 2018-10-11 13:45:07.773361 | root[root] @ localhost [::1] | 1651 | 1 | Query | select andauth from phome_enewsuserloginck where userid='1' | | 2018-10-11 13:45:07.772358 | root[root] @ localhost [::1] | 1651 | 1 | Query | SET character_set_connection=utf8,character_set_results=utf8,character_set_client=binary,sql_mode='' | | 2018-10-11 13:45:07.772358 | root[root] @ localhost [::1] | 1651 | 1 | Init DB | empirecms | | 2018-10-11 13:45:07.768348 | [root] @ localhost [::1] | 1651 | 1 | Connect | [email protected] on using TCP/IP

我們剔除掉一些SELECT語句和無用的系統相關的SET語句,只留下update和insert語句,最後經過排查,剩下這些操作還是有用的,然後在使用下面語句生成的記錄上進行刪除,修改操作,都是正常的,因此可以斷定,只需要對如下表進行操作, 就可以完成新增記錄的操作。

1.insert into phome_ecms_news_index(classid,checked,newstime,truetime,lastdotime,havehtml) values('2','1','1539238638','1539238707','1539238707','1'); 2.insert into phome_ecms_news_data_1(id,classid,keyid,dokey,newstempid,closepl,haveaddfen,infotags,writer,befrom,newstext) values('4','2','','1','0','0',0,'','','','<p>&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content</p>') 3.insert into phome_ecms_news(id,classid,ttid,onclick,plnum,totaldown,newspath,filename,userid,username,firsttitle,isgood,ispic,istop,isqf,ismember,isurl,truetime,lastdotime,havehtml,groupid,userfen,titlefont,titleurl,stb,fstb,restb,keyboard,title,ftitle,newstime,titlepic,smalltext) values('4','2','0','0',0,'0','2018-10-11','','1','admin','0','0','0','0','0',0,'0','1539238707','1539238707','1','0','0','','','1','1','1','keyword','title','fubiao','1539238638','','content desc') 4.update phome_enewsclass set allinfos=allinfos+1,infos=infos+1 where classid='2' limit 1 5.update phome_ecms_news set filename='4',titleurl='/youhua/seo/2018-10-11/4.html' where id='4'6.update phome_enewsfile_1 set pubid='1000010000000002',classid='2',id='2',cjid=0 where cjid='1539236638' 7.update phome_enewspublic_update set lastnuminfo=lastnuminfo+1,lastnuminfotb='|1,40|',todaynuminfo=todaynuminfo+1 limit 18.insert into phome_enewsdolog(username,logip,logtime,enews,doing,pubid,ipport) values('admin','::1','2018-10-11 13:45:07','AddNews','classid=2<br>id=2<br>title=title','1000010000000002','63863')

使用過程中發現:update phome_enewsfile_1 set pubid='1000010000000002',classid='2',id='2',cjid=0 where cjid='1539236638' 這個其實也是一條無效操作,因為這個表始終是空的,後續會不會在某些時候有用,暫時還不得而知,先去掉這條語句,後來經過研究發現這條記錄是用於儲存附件的, 因為暫時沒有附件,所以可以先不處理這條記錄。 最後一條是日誌記錄,經過測試發現日誌是可以不寫的,不會影響操作的正常進行,所以可以先忽略。

這樣最後有用的剩下六條:1.insert into phome_ecms_news_index(classid,checked,newstime,truetime,lastdotime,havehtml) values('2','1','1539238638','1539238707','1539238707','1');2.insert into phome_ecms_news_data_1(id,classid,keyid,dokey,newstempid,closepl,haveaddfen,infotags,writer,befrom,newstext) values('4','2','','1','0','0',0,'','','','<p>&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content</p>')3.insert into phome_ecms_news(id,classid,ttid,onclick,plnum,totaldown,newspath,filename,userid,username,firsttitle,isgood,ispic,istop,isqf,ismember,isurl,truetime,lastdotime,havehtml,groupid,userfen,titlefont,titleurl,stb,fstb,restb,keyboard,title,ftitle,newstime,titlepic,smalltext) values('4','2','0','0',0,'0','2018-10-11','','1','admin','0','0','0','0','0',0,'0','1539238707','1539238707','1','0','0','','','1','1','1','keyword','title','fubiao','1539238638','','content desc')4.update phome_enewsclass set allinfos=allinfos+1,infos=infos+1 where classid='2' limit 15.update phome_ecms_news set filename='4',titleurl='/youhua/seo/2018-10-11/4.html' where id='4'6.update phome_enewspublic_update set lastnuminfo=lastnuminfo+1,lastnuminfotb='|1,40|',todaynuminfo=todaynuminfo+1 limit 1

我們只需要在匯入資料庫時做好這些操作就可以了。

insert into phome_ecms_news_index(classid,checked,newstime,truetime,lastdotime,havehtml) values('2','1','1539238638','1539238707','1539238707','1');insert into phome_ecms_news_data_1(id,classid,keyid,dokey,newstempid,closepl,haveaddfen,infotags,writer,befrom,newstext) values('4','2','','1','0','0',0,'','','','<p>&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content</p>')insert into phome_ecms_news(id,classid,ttid,onclick,plnum,totaldown,newspath,filename,userid,username,firsttitle,isgood,ispic,istop,isqf,ismember,isurl,truetime,lastdotime,havehtml,groupid,userfen,titlefont,titleurl,stb,fstb,restb,keyboard,title,ftitle,newstime,titlepic,smalltext) values('4','2','0','0',0,'0','2018-10-11','','1','admin','0','0','0','0','0',0,'0','1539238707','1539238707','1','0','0','','','1','1','1','keyword','title','fubiao','1539238638','','content desc')update phome_enewsclass set allinfos=allinfos+1,infos=infos+1 where classid='2' limit 1update phome_ecms_news set filename='@@IDENTITY',titleurl=concat('/youhua/seo/2018-10-11/',@@IDENTITY,'.html') where [email protected]@IDENTITYupdate phome_enewspublic_update set lastnuminfo=lastnuminfo+1,lastnuminfotb='|1,40|',todaynuminfo=todaynuminfo+1 limit 1

為了讓文章ID唯一,我們需要使用@@IDENTITY識別符號,來實現唯一標識的傳遞,那麼就不能直接用上面的順序來進行資料庫填寫,因為@@IDENTITY(@@identity是表示的是最近一次向具有identity屬性(即自增列)的表插入資料時對應的自增列的值,是系統定義的全域性變數。)insert into phome_ecms_news_index(classid,checked,newstime,truetime,lastdotime,havehtml) values('2','1','1539238888','1539238888','1539238888','1');insert into phome_ecms_news(id,classid,ttid,onclick,plnum,totaldown,newspath,filename,userid,username,firsttitle,isgood,ispic,istop,isqf,ismember,isurl,truetime,lastdotime,havehtml,groupid,userfen,titlefont,titleurl,stb,fstb,restb,keyboard,title,ftitle,newstime,titlepic,smalltext) values('6','2','0','0',0,'0','2018-10-11','','1','admin','0','0','0','0','0',0,'0','1539238888','1539238888','1','0','0','','','1','1','1','keyword','title','fubiao','1539238888','','content desc')update phome_ecms_news set filename='6',titleurl=concat('/youhua/seo/2018-10-11/6.html') where id=6insert into phome_ecms_news_data_1(id,classid,keyid,dokey,newstempid,closepl,haveaddfen,infotags,writer,befrom,newstext) values('6','2','','1','0','0',0,'','','','<p>&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content&nbsp;content</p>')update phome_enewsclass set allinfos=allinfos+1,infos=infos+1 where classid='2' limit 1;update phome_enewspublic_update set lastnuminfo=lastnuminfo+1,lastnuminfotb='|1,43|',todaynuminfo=todaynuminfo+1 limit 1;

把部分內容替換成系統標識以後,如下所示:insert into phome_ecms_news_index(classid,checked,newstime,truetime,lastdotime,havehtml) values('2','1','[系統時間戳]','[系統時間戳]','[系統時間戳]','1')

insert into phome_ecms_news(id,classid,ttid,onclick,plnum,totaldown,newspath,filename,userid,username,firsttitle,isgood,ispic,istop,isqf,ismember,isurl,truetime,lastdotime,havehtml,groupid,userfen,titlefont,titleurl,stb,fstb,restb,keyboard,title,ftitle,newstime,titlepic,smalltext) values(@@IDENTITY,'2','0','0',0,'0','2018-10-11','','1','admin','0','0','0','0','0',0,'0','[系統時間戳]','[系統時間戳]','1','0','0','','','1','1','1','keyword','[標籤:標題]','','[系統時間戳]','','');

update phome_ecms_news set filename='@@IDENTITY',titleurl=concat('/youhua/seo/2018-10-11/',@@IDENTITY,'.html') where [email protected]@IDENTITY;

insert into phome_ecms_news_data_1(id,classid,keyid,dokey,newstempid,closepl,haveaddfen,infotags,writer,befrom,newstext) values(@@IDENTITY,'2','','1','0','0',0,'','','','[標籤:內容]');

update phome_enewsclass set allinfos=allinfos+1,infos=infos+1 where classid='2' limit 1;

update phome_enewspublic_update set lastnuminfo=lastnuminfo+1,lastnuminfotb=concat('|1,',lastnuminfo,'|'),todaynuminfo=todaynuminfo+1 limit 1;

具體操作如下圖:

通過測試釋出,檢視釋出後的情況,可以看到已經發布的記錄:

更新記錄,生成html檔案後,開啟如下圖所示:

這樣火車頭的內容釋出以資料庫形式釋出,到這裡就結束了。上面這五條sql語句,是資料庫釋出必備的,整個過程斷斷續續除錯了幾天,發現火車頭採集器跟帝國的文章釋出相當繁瑣, 而且帝國CMS的釋出好像用的人不是很多,寫出來共享給大家說不定有用,過程相當不易,覺得有用的朋友們點個贊。。。