Java資料爬取——爬取攜程酒店資料（二）
阿新 • • 發佈：2019-01-22
1.首先思考怎樣根據地域獲取地域酒店資訊，那麼我們看一下攜程上是怎樣獲得的。
還是開啟http://hotels.ctrip.com/domestic-city-hotel.html 這個地址，隨便點選一個地區進去（這裡我選取澳門作為示例），點選第二頁資料，很高興發現
2.模擬請求，獲取資料
從這裡可以發現，是POST請求，請求的body中有很多引數，
所以先封裝一下引數，經過測試，有些引數可以不用傳遞，這裡寫了一個方法，模擬請求
public String getHotelListString(HotelCity city, String page){
HashMap<String, String> params = new 說明：這個方法對返回結果進行了操作，直接把hotelPositionJSON和hotelAmount的內容提取了出來（因為直接轉換時發現數據中含有轉義符導致轉為JSON失敗），操作後的資料格式為：
{
"hotelAmount": 11265,
"hotelPositionJSON": [
{
"id": "6297824",
"name": "北京浣川招待所",
"lat": "39.918086",
"lon": "116.427508",
"url": "/hotel/6297824.html?isFull=F#ctm_ref=hod_sr_map_dl_txt_1",
"img": "http://pic.c-ctrip.com/hotels110127/hotel_example.jpg",
"address": "東城區東城區外交部街46號-1。 （ 北京站、建國門地區）",
"score": "0.0",
"dpscore": "0",
"dpcount": "0",
"star": "hotel_diamond01",
"stardesc": "攜程使用者評定為1鑽",
"shortName": "",
"isSingleRec": "false"
},
{
"id": "6298279",
"name": "北京懷柔喇叭溝門海燕農家院",
"lat": "40.956471",
"lon": "116.513108",
"url": "/hotel/6298279.html?isFull=F#ctm_ref=hod_sr_map_dl_txt_2",
"img": "http://pic.c-ctrip.com/hotels110127/hotel_example.jpg",
"address": "懷柔區黃甸子村。 （ 懷柔風景區）",
"score": "0.0",
"dpscore": "0",
"dpcount": "0",
"star": "hotel_diamond01",
"stardesc": "攜程使用者評定為1鑽",
"shortName": "",
"isSingleRec": "false"
}
]
}3.迴圈遍歷城市資料，查詢每個城市的酒店即可
long startTime = System.currentTimeMillis();
HotelCitySpider citySpider = new HotelCitySpider();
HotelSpider spider = new HotelSpider();
List<HotelCity> cities = citySpider.getHotelCities();
long getCityTime = System.currentTimeMillis();
System.out.println("獲取城市所用時間(ms):" + (getCityTime - startTime));
spider.createTable();
for (HotelCity hotelCity : cities) {
spider.saveHotels(hotelCity, spider.getHotelList(hotelCity));
}
long saveHotelTime = System.currentTimeMillis();
System.out.println("獲取酒店並存儲所用時間(ms):" + (saveHotelTime - startTime));這裡我仍然將資料儲存到資料庫中
4.批量插入資料
/**
* 儲存每個城市的酒店列表
* @param city
* @param hotels
*/
public void saveHotels(HotelCity city, List<Hotel> hotels) {
for (Hotel hotel : hotels) {
StringBuffer insert_sql = new StringBuffer();
insert_sql.append("insert into ctrip_hotel "
+ "(hotel_id, city_id, city_name, name, lat, lon, url, img, address, score, dpscore, dpcount, star, stardesc, shortName, isSingleRec) values (");
insert_sql.append("'" + hotel.getId() + "'");
insert_sql.append(", " + city.getCityId());
insert_sql.append(", '" + city.getCityName() + "'");
insert_sql.append(", '" + hotel.getName() + "'");
insert_sql.append(", " + hotel.getLat());
insert_sql.append(", " + hotel.getLon());
insert_sql.append(", '" + hotel.getUrl() + "'");
insert_sql.append(", '" + hotel.getImg() + "'");
insert_sql.append(", '" + hotel.getAddress() + "'");
insert_sql.append(", " + hotel.getScore());
insert_sql.append(", " + hotel.getDpscore());
insert_sql.append(", " + hotel.getDpcount());
insert_sql.append(", '" + hotel.getStar() + "'");
insert_sql.append(", '" + hotel.getStardesc() + "'");
insert_sql.append(", '" + hotel.getShortName() + "'");
insert_sql.append(", " + hotel.getIsSingleRec() + ")");
try {
preparedStatement = conn.prepareStatement(insert_sql.toString());
preparedStatement.execute();
} catch (Exception e) {
e.getMessage();
continue;
}
}
}原本想批量插入，但是資料有重複的hotel_id，preparedStatement.executeBatch()會報錯，所以還是一條條插入。
github原始碼地址 https://github.com/jianiuqi/CTripSpider