基於Python的大資料的分頁模型程式碼
阿新 • • 發佈：2018-11-21
最近在寫一個cmdb系統的分頁，儘管Django本身有分頁的模組兒，但是還是想實現一下自己心中想的分頁的一種邏輯
因為，在我們工作當中，當我們的資料量超級大的時候，其實我們每次分頁查詢都不必將所有的資料查詢出來，而是可以按階段的查詢，舉個例子
每次查詢5頁資料，當需要第六頁的時候，再次進行載入,為了更加明瞭，做了舉例
首先我們規定好每次查詢5頁，每頁4條資料
1 db_data = list(range(1,100)) 2 3 final_page = len(db_data) 4 5 one_page_num = 5 #每頁4條 6 one_time_num = 5 #每次查5頁
1、查詢索引的邏輯 設定findIndex查詢的索引
1 # page = 1 int(1/5) + 1 findIndex = 1 2 # page = 2 int(2/5) + 1 findIndex = 1 3 # page = 3 int(3/5) + 1 findIndex = 1 4 # page = 4 int(4/5) + 1 findIndex = 1 5 # page = 5 int(5/5) findIndex = 1 6 # page = 6 int(6/5) + 1 findIndex = 27 # page = 7 int(7/5) + 1 findIndex = 2 8 # page = 8 int(8/5) + 1 findIndex = 2 9 # page = 9 int(9/5) + 1 findIndex = 2 10 # page = 10 int(10/5) findIndex = 2 11 # page = 11 int(11/5) + 1 findIndex = 3 12 # page = 12 int(12/5) + 1 findIndex = 3 13 # page = 13 int(13/5) + 1 findIndex = 314 # page = 14 int(14/5) + 1 findIndex = 3 15 # page = 15 int(15/5) findIndex = 3
所以我們可以寫如下程式碼
1 if page/one_time_num > int(page/one_time_num): 2 findIndex = int(page / one_time_num) + 1 3 else: 4 findIndex = int(page / one_time_num) 5 #進行第一次查詢，查詢每次5頁，每頁4條的資料 6 select_num = one_time_num * one_page_num #這裡決一次定查詢20條 7 select_start = (findIndex - 1) * select_num # 開始查詢的索引 8 select_down = findIndex * select_num # 結束查詢的索引 9 select_data = db_data[select_start: select_down] #這裡查詢出了一次20條資料
這個時候是查詢出20條資料了，但是，我們想要的具體的資料和頁碼對不上，所以做了以下的資料
1 #下面對20條資料進行擷取 2 #findIndex 1 3 # page = 1 1-(0*5) gnPage = 1 4 # page = 2 2-(0*5) gnPage = 2 5 # page = 3 3-(0*5) gnPage = 3 6 # page = 4 4-(0*5) gnPage = 4 7 # page = 5 5-(0*5) gnPage = 5 8 #findIndex 2 9 # page = 6 6-(1*5) gnPage = 1 10 # page = 7 7-(1*5) gnPage = 2 11 # page = 8 8-(1*5) gnPage = 3 12 # page = 9 9-(1*5) gnPage = 4 13 # page = 10 10-(1*5) gnPage = 5 14 #findIndex 3 15 # page = 11 11-(2*5) gnPage = 1 16 # page = 12 12-(2*5) gnPage = 2 17 # page = 13 13-(2*5) gnPage = 3 18 # page = 14 14-(2*5) gnPage = 4 19 # page = 15 15-(2*5) gnPage = 5
所以我們可以得到以下程式碼來獲取具體的一頁資料
#設定 擷取的索引位now_index now_index = page - (findIndex-1)*one_time_num #設定單頁的擷取起始 page_start = (now_index - 1) * one_page_num page_end = now_index * one_page_num #開始擷取 page_data = select_data[page_start:page_end]
最後進行程式碼結合
1 #coding:utf-8 2 3 db_data = list(range(1,100)) 4 5 one_page_num = 5 #每頁4條 6 one_time_num = 5 #每次查5頁 7 8 9 while True: 10 page = int(input("page >>>")) #輸入頁碼 11 12 #查詢索引的邏輯 設定findIndex查詢的索引 13 # page = 1 int(1/5) + 1 findIndex = 1 14 # page = 2 int(2/5) + 1 findIndex = 1 15 # page = 3 int(3/5) + 1 findIndex = 1 16 # page = 4 int(4/5) + 1 findIndex = 1 17 # page = 5 int(5/5) findIndex = 1 18 # page = 6 int(6/5) + 1 findIndex = 2 19 # page = 7 int(7/5) + 1 findIndex = 2 20 # page = 8 int(8/5) + 1 findIndex = 2 21 # page = 9 int(9/5) + 1 findIndex = 2 22 # page = 10 int(10/5) findIndex = 2 23 # page = 11 int(11/5) + 1 findIndex = 3 24 # page = 12 int(12/5) + 1 findIndex = 3 25 # page = 13 int(13/5) + 1 findIndex = 3 26 # page = 14 int(14/5) + 1 findIndex = 3 27 # page = 15 int(15/5) findIndex = 3 28 29 if page/one_time_num > int(page/one_time_num): 30 findIndex = int(page / one_time_num) + 1 31 else: 32 findIndex = int(page / one_time_num) 33 #進行第一次查詢，查詢每次5頁，每頁4條的資料 34 select_num = one_time_num * one_page_num #這裡決一次定查詢20條 35 select_start = (findIndex - 1) * select_num # 開始查詢的索引 36 select_down = findIndex * select_num # 結束查詢的索引 37 38 select_data = db_data[select_start: select_down] #這裡查詢出了一次20條資料 39 40 #下面對20條資料進行擷取 41 #findIndex 1 42 # page = 1 1-(0*5) gnPage = 1 43 # page = 2 2-(0*5) gnPage = 2 44 # page = 3 3-(0*5) gnPage = 3 45 # page = 4 4-(0*5) gnPage = 4 46 # page = 5 5-(0*5) gnPage = 5 47 #findIndex 2 48 # page = 6 6-(1*5) gnPage = 1 49 # page = 7 7-(1*5) gnPage = 2 50 # page = 8 8-(1*5) gnPage = 3 51 # page = 9 9-(1*5) gnPage = 4 52 # page = 10 10-(1*5) gnPage = 5 53 #findIndex 3 54 # page = 11 11-(2*5) gnPage = 1 55 # page = 12 12-(2*5) gnPage = 2 56 # page = 13 13-(2*5) gnPage = 3 57 # page = 14 14-(2*5) gnPage = 4 58 # page = 15 15-(2*5) gnPage = 5 59 #設定 擷取的索引位now_index 60 now_index = page - (findIndex-1)*one_time_num 61 #設定單頁的擷取起始 62 page_start = (now_index - 1) * one_page_num 63 page_end = now_index * one_page_num 64 #開始擷取 65 page_data = select_data[page_start:page_end] 66 #輸出效果 67 print("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++") 68 print("當前是第%s次查詢"%findIndex) 69 print("當前查詢到的所有資料是 \n %s" % select_data) 70 print("當前是 %s 頁" % page) 71 print("當前的資料是 \n %s " % page_data) 72 print("+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++")
大家試一下吧。