比較器、過濾器、過濾器的操作符
阿新 • • 發佈：2019-01-03
過濾器的操作符
LESS <
LESS_OR_EQUAL <=
EQUAL =
NOT_EQUAL <>
GREATER_OR_EQUAL >=
GREATER >
NO_OP no operation比較器
BinaryComparator 按位元組索引順序比較指定位元組陣列，採用Bytes.compareTo(byte[])
BinaryPrefixComparator 跟前面相同，只是比較左端的資料是否相同
NullComparator 判斷給定的是否為空
BitComparator 按位比較 a BitwiseOp class 做異或，與，並操作
RegexStringComparator 提供一個正則的比較器，僅支援 EQUAL 和非EQUAL
SubstringComparator 判斷提供的子串是否出現在table的value Hbase的過濾器分類
1.比較過濾器 Comparison Filters
1.1 RowFilter
建構函式：
public RowFilter(org.apache.hadoop.hbase.filter.CompareFilter.CompareOp rowCompareOp, org.apache.hadoop.hbase.filter.WritableByteArrayComparable rowComparator) {}
//選擇比較RowKey來確認返回訊息示例程式碼：
public class RowFilterExample {
public 1.2 FamilyFilter
建構函式
public FamilyFilter(CompareOp familyCompareOp, WritableByteArrayComparable familyComparator) {} 示例程式碼
//通過對比FamilyKey去獲取資料
public class FamilyFilterExample {
public static void main(String[] args) throws IOException {
Configuration conf = HBaseConfiguration.create();
HBaseHelper helper = HBaseHelper.getHelper(conf);
helper.dropTable("testtable");
helper.createTable("testtable", "colfam1", "colfam2", "colfam3", "colfam4");
System.out.println("Adding rows to table...");
helper.fillTable("testtable", 1, 10, 2, "colfam1", "colfam2", "colfam3", "colfam4");
HTable table = new HTable(conf, "testtable");
// vv FamilyFilterExample
Filter filter1 = new FamilyFilter(CompareFilter.CompareOp.LESS, // co FamilyFilterExample-1-Filter Create filter, while specifying the comparison operator and comparator.
new BinaryComparator(Bytes.toBytes("colfam3")));
Scan scan = new Scan();
scan.setFilter(filter1);
ResultScanner scanner = table.getScanner(scan); // co FamilyFilterExample-2-Scan Scan over table while applying the filter.
// ^^ FamilyFilterExample
System.out.println("Scanning table... ");
// vv FamilyFilterExample
for (Result result : scanner) {
System.out.println(result);
}
scanner.close();
Get get1 = new Get(Bytes.toBytes("row-5"));
get1.setFilter(filter1);
Result result1 = table.get(get1); // co FamilyFilterExample-3-Get Get a row while applying the same filter.
System.out.println("Result of get(): " + result1);
Filter filter2 = new FamilyFilter(CompareFilter.CompareOp.EQUAL,
new BinaryComparator(Bytes.toBytes("colfam3")));
Get get2 = new Get(Bytes.toBytes("row-5")); // co FamilyFilterExample-4-Mismatch Create a filter on one column family while trying to retrieve another.
get2.addFamily(Bytes.toBytes("colfam1"));
get2.setFilter(filter2);
Result result2 = table.get(get2); // co FamilyFilterExample-5-Get2 Get the same row while applying the new filter, this will return "NONE".
System.out.println("Result of get(): " + result2);
// ^^ FamilyFilterExample
}
} 1.3QualifierFilter
建構函式
public QualifierFilter(CompareOp qualifierCompareOp, WritableByteArrayComparable qualifierComparator) { } 示例程式碼：
//通過和列名比較，返回為真的資料
// vv QualifierFilterExample
Filter filter = new QualifierFilter(CompareFilter.CompareOp.LESS_OR_EQUAL,
new BinaryComparator(Bytes.toBytes("col-2")));
Scan scan = new Scan();
scan.setFilter(filter);
ResultScanner scanner = table.getScanner(scan);
// ^^ QualifierFilterExample
System.out.println("Scanning table... ");
// vv QualifierFilterExample
for (Result result : scanner) {
System.out.println(result);
}
scanner.close();
Get get = new Get(Bytes.toBytes("row-5"));
get.setFilter(filter);
Result result = table.get(get);
System.out.println("Result of get(): " + result); 1.4 ValueFliter
public ValueFilter(CompareOp valueCompareOp, WritableByteArrayComparable valueComparator) { }//對比列值獲取返回資料
Filter filter = new ValueFilter(CompareFilter.CompareOp.EQUAL, // co ValueFilterExample-1-Filter Create filter, while specifying the comparison operator and comparator.
new SubstringComparator(".4") );
Scan scan = new Scan();
scan.setFilter(filter); // co ValueFilterExample-2-SetFilter Set filter for the scan.
ResultScanner scanner = table.getScanner(scan);
// ^^ ValueFilterExample
System.out.println("Results of scan:");
// vv ValueFilterExample
for (Result result : scanner) {
for (KeyValue kv : result.raw()) {
System.out.println("KV: " + kv + ", Value: " + // co ValueFilterExample-3-Print1 Print out value to check that filter works.
Bytes.toString(kv.getValue()));
}
}
scanner.close();
Get get = new Get(Bytes.toBytes("row-5"));
get.setFilter(filter); // co ValueFilterExample-4-SetFilter2 Assign same filter to Get instance.
Result result = table.get(get);
// ^^ ValueFilterExample
System.out.println("Result of get: ");
// vv ValueFilterExample
for (KeyValue kv : result.raw()) {
System.out.println("KV: " + kv + ", Value: " +
Bytes.toString(kv.getValue()));
} 1.5 DependentColumnFilter
該過濾器有兩個引數 —— 列族和列修飾。 嘗試找到該列所在的每一行，並返回該行具有相同時間戳的全部鍵值對。如果某一行不包含指定的列，則該行的任何鍵值對都不返回。
該過濾器還可以有一個可選布林引數 —— dropDependentColumn. 如果為true, 從屬的列不返回。
該過濾器還可以有兩個可選引數 —— 一個比較操作符和一個值比較器，用於列族和修飾的進一步檢查。如果從屬的列找到，其值還必須通過值檢查，然後就是時間戳必須考慮。示例程式碼
package filters;
// cc DependentColumnFilterExample Example using a filter to include only specific column families
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.filter.BinaryPrefixComparator;
import org.apache.hadoop.hbase.filter.CompareFilter;
import org.apache.hadoop.hbase.filter.DependentColumnFilter;
import org.apache.hadoop.hbase.filter.Filter;
import org.apache.hadoop.hbase.filter.RegexStringComparator;
import org.apache.hadoop.hbase.filter.WritableByteArrayComparable;
import org.apache.hadoop.hbase.util.Bytes;
import util.HBaseHelper;
import java.io.IOException;
public class DependentColumnFilterExample {
private static HTable table = null;
// vv DependentColumnFilterExample
private static void filter(boolean drop,
CompareFilter.CompareOp operator,
WritableByteArrayComparable comparator)
throws IOException {
Filter filter;
if (comparator != null) {
filter = new DependentColumnFilter(Bytes.toBytes("colfam1"), // co DependentColumnFilterExample-1-CreateFilter Create the filter with various options.
Bytes.toBytes("col-5"), drop, operator, comparator);
} else {
filter = new DependentColumnFilter(Bytes.toBytes("colfam1"),
Bytes.toBytes("col-5"), drop);
}
Scan scan = new Scan();
scan.setFilter(filter);
ResultScanner scanner = table.getScanner(scan);
// ^^ DependentColumnFilterExample
System.out.println("Results of scan:");
// vv DependentColumnFilterExample
for (Result result : scanner) {
for (KeyValue kv : result.raw()) {
System.out.println("KV: " + kv + ", Value: " +
Bytes.toString(kv.getValue()));
}
}
scanner.close();
Get get = new Get(Bytes.toBytes("row-5"));
get.setFilter(filter);
Result result = table.get(get);
// ^^ DependentColumnFilterExample
System.out.println("Result of get: ");
// vv DependentColumnFilterExample
for (KeyValue kv : result.raw()) {
System.out.println("KV: " + kv + ", Value: " +
Bytes.toString(kv.getValue()));
}
// ^^ DependentColumnFilterExample
System.out.println("");
// vv DependentColumnFilterExample
}
public static void main(String[] args) throws IOException {
// ^^ DependentColumnFilterExample
Configuration conf = HBaseConfiguration.create();
HBaseHelper helper = HBaseHelper.getHelper(conf);
helper.dropTable("testtable");
helper.createTable("testtable", "colfam1", "colfam2");
System.out.println("Adding rows to table...");
helper.fillTable("testtable", 1, 10, 10, true, "colfam1", "colfam2");
table = new HTable(conf, "testtable");
// vv DependentColumnFilterExample
filter(true, CompareFilter.CompareOp.NO_OP, null);
filter(false, CompareFilter.CompareOp.NO_OP, null); // co DependentColumnFilterExample-2-Filter Call filter method with various options.
filter(true, CompareFilter.CompareOp.EQUAL,
new BinaryPrefixComparator(Bytes.toBytes("val-5")));
filter(false, CompareFilter.CompareOp.EQUAL,
new BinaryPrefixComparator(Bytes.toBytes("val-5")));
filter(true, CompareFilter.CompareOp.EQUAL,
new RegexStringComparator(".*\\.5"));
filter(false, CompareFilter.CompareOp.EQUAL,
new RegexStringComparator(".*\\.5"));
}
// ^^ DependentColumnFilterExample} 2. Dedicated Filter
2.1 SingleColumValueFilter
//選定列簇和某一列，然後與列的value相比，正確的返回全部的row，注意如果某一行不含有該列，同樣返回，除非通過filterIfColumnMissing 設定成真。
//第一個建構函式相當於構建了一個BinaryComparator的例項。其他的跟CompareFilter的引數含義一樣。
SingleColumnValueFilter(byte[] family, byte[] qualifier, CompareOp compareOp, byte[] value){}
SingleColumnValueFilter(byte[] family, byte[] qualifier, CompareOp compareOp, WritableByteArrayComparable comparator){ }
//第一個建構函式相當於構建了一個BinaryComparator的例項。其他的跟CompareFilter的引數含義一樣。
boolean getFilterIfMissing()
void setFilterIfMissing(boolean filterIfMissing)
boolean getLatestVersionOnly()
void setLatestVersionOnly(boolean latestVersionOnly)
//如果 filterIfColumnMissing 標誌設為真，如果該行沒有指定的列，那麼該行的所有列將不發出。預設值為假。
//如果 setLatestVersionOnly 標誌設為假，將檢查此前的版本。預設值為真。例項如下：
// vv SingleColumnValueFilterExample
SingleColumnValueFilter filter = new SingleColumnValueFilter(
Bytes.toBytes("colfam1"),
Bytes.toBytes("col-5"),
CompareFilter.CompareOp.NOT_EQUAL,
new SubstringComparator("val-5"));
filter.setFilterIfMissing(true);
Scan scan = new Scan();
scan.setFilter(filter);
ResultScanner scanner = table.getScanner(scan);
// ^^ SingleColumnValueFilterExample
System.out.println("Results of scan:");
// vv SingleColumnValueFilterExample
for (Result result : scanner) {
for (KeyValue kv : result.raw()) {
System.out.println("KV: " + kv + ", Value: " +
Bytes.toString(kv.getValue()));
}
}
scanner.close();
Get get = new Get(Bytes.toBytes("row-6"));
get.setFilter(filter);
Result result = table.get(get);
System.out.println("Result of get: ");
for (KeyValue kv : result.raw()) {
System.out.println("KV: " + kv + ", Value: " +
Bytes.toString(kv.getValue()));
} 2.2 SingleColumnValueExcludeFilter
與SingleColumnValueFilter相反，與條件相符的將不會返回
2.2 PrefixFilter
所有的row的例項匹配prefix的時候返回結果集合
Filter filter = new PrefixFilter(Bytes.toBytes("row1"));
Scan scan = new Scan();
scan.setFilter(filter);
ResultScanner scanner = table.getScanner(scan);
for(Result result: scanner){
for(KeyValue kv: result.raw()) {
System.out.println("KV:" + kv + ", Value:" + Bytes.toString(kv.getValue()));
}
}
scanner.close();
Get get = new Get(Bytes.toBytes("row-5"));
get.setFilter(filter);
Result result = table.get(get);
for(KeyValue kv : result.raw()){
System.out.println("KV:" + kv + ", Value:" + Bytes.toString(kv.getValue()));
} 2.4 PageFilter
//頁過濾
//通過設定pagesize可以設定返回每一頁的page大小
//客戶端需要記錄上一次返回的row的Key值
package hbaseTest;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.filter.Filter;
import org.apache.hadoop.hbase.filter.PageFilter;
import org.apache.hadoop.hbase.util.Bytes;
import java.io.IOException;
/**
* Hello world!
*/
public class PageFilterExample {
public static void main(String[] args) throws IOException {
Configuration config = HBaseConfiguration.create();
config.set("hbase.zookeeper.quorum", "QT-H-0038");
String tableName = "testTable";
String cfName = "colfam1";
final byte[] POSTFIX = new byte[] { 0x00 };
HTable table = new HTable(config, tableName);
Filter filter = new PageFilter(15);
byte[] lastRow = null;
int totalRows = 0;
while (true) {
Scan scan = new Scan();
scan.setFilter(filter);
if(lastRow != null){
//注意這裡添加了POSTFIX操作，不然死迴圈了
byte[] startRow = Bytes.add(lastRow,POSTFIX);
scan.setStartRow(startRow);
}
ResultScanner scanner = table.getScanner(scan);
int localRows = 0;
Result result;
while((result = scanner.next()) != null){
System.out.println(localRows++ + ":" + result);
totalRows ++;
lastRow = result.getRow();
}
scanner.close();
if(localRows == 0) break;
}
System.out.println("total rows:" + totalRows);
}
}
//因為hbase的row是字典序列排列的，因此上一次的lastrow需要新增額外的0(0x00)表示新的開始。另外startKey的那一行是包含在scan裡面的。
// final byte[] POSTFIX = new byte[] { 0x00 }; 2.5 KeyOnlyFilter
因為一些應用只想獲取data資料，而不是真實的val，可以使用這個過濾器。該過濾器通過
KeyOnlyFilter(boolean lenAsVal)
//lenAsVal預設為假，表示不把val的長度作為val。否則 val的長度將作為val輸出。
final byte[] POSTFIX = new byte[] { 0x00 };
HTable table = new HTable(config, tableName);
Filter filter = new KeyOnlyFilter(false);
byte[] lastRow = null;
int totalRows = 0;
Scan scan = new Scan();
scan.setFilter(filter);
ResultScanner scanner = table.getScanner(scan);
for(Result result: scanner){
for(KeyValue kv: result.raw()){
System.out.println(kv + ":" + Bytes.toString(kv.getValue()));
}
} 2.6 FirstKeyOnlyFilter
//在對hbase的表進行掃描的時候，如果指定了FirstKeyOnlyFilter過濾條件則僅僅會返回相同key的第一條kv。
//當對hbase中的表進行count，sum操作等集合操作的時候，使用FirstKeyOnlyFilter會帶來效能上的提升。
public class KeyOnlyFilterExample {
public static void main(String[] args) throws IOException {
Configuration config = HBaseConfiguration.create();
config.set("hbase.zookeeper.quorum", "QT-H-0038");
String tableName = "testTable";
String cfName = "colfam1";
final byte[] POSTFIX = new byte[] { 0x00 };
HTable table = new HTable(config, tableName);
Filter filter = new FirstKeyOnlyFilter();
byte[] lastRow = null;
int totalRows = 0;
Scan scan = new Scan();
scan.setFilter(filter);
ResultScanner scanner = table.getScanner(scan);
for(Result result: scanner){
for(KeyValue kv: result.raw()){
System.out.println(kv + ":" + Bytes.toString(kv.getValue()));
}
}
}
}
返回的結果是
row-5/colfam1:qual1/1354673733503/Put/vlen=4:row1
row1/colfam1:qual1/1354432930568/Put/vlen=4:val1
row2/colfam1:qual2/1354432930568/Put/vlen=4:val3
如果註釋掉過濾器的返回的結果是：
row-5/colfam1:qual1/1354673733503/Put/vlen=4:row1
row1/colfam1:qual1/1354432930568/Put/vlen=4:val1
row1/colfam1:qual2/1354435819120/Put/vlen=4:val2
row2/colfam1:qual2/1354432930568/Put/vlen=4:val3 2.7 InclusiveStopFilter
//因為hbase的scan包含start-row不包含stop-row 如果使用這個過濾器我們可以包含stop-row
HTable table = new HTable(config, tableName);
Filter filter = new InclusiveStopFilter(Bytes.toBytes("row1"));
Scan scan = new Scan();
scan.setFilter(filter);
scan.setStartRow(Bytes.toBytes("row-5"));
ResultScanner scanner = table.getScanner(scan);
for(Result result: scanner){
System.out.println(result);
}
//會看到row1包含在結果中了。2.8 TimestampsFilter
當訪問某個Timestamp的新聞的時候，我們需要如下的程式碼：
TimestampsFilter(List<Long> timestamps)
接受的引數的list引數，該Filter也可以和scan.setTimeRange混合使用。例如：
// vv TimestampFilterExample
List<Long> ts = new ArrayList<Long>();
ts.add(new Long(5));
ts.add(new Long(10)); // co TimestampFilterExample-1-AddTS Add timestamps to the list.
ts.add(new Long(15));
Filter filter = new TimestampsFilter(ts);
Scan scan1 = new Scan();
scan1.setFilter(filter); // co TimestampFilterExample-2-AddFilter Add the filter to an otherwise default Scan instance.
ResultScanner scanner1 = table.getScanner(scan1);
// ^^ TimestampFilterExample
System.out.println("Results of scan #1:");
// vv TimestampFilterExample
for (Result result : scanner1) {
System.out.println(result);
}
scanner1.close();
Scan scan2 = new Scan();
scan2.setFilter(filter);
scan2.setTimeRange(8, 12); // co TimestampFilterExample-3-AddTSRange Also add a time range to verify how it affects the filter
ResultScanner scanner2 = table.getScanner(scan2);
// ^^ TimestampFilterExample
System.out.println("Results of scan #2:");
// vv TimestampFilterExample
for (Result result : scanner2) {
System.out.println(result);
}
scanner2.close(); 2.9 ColumnCountGetFilter
//在scan時是無用的2.10 ColumnPaginationFilter（下來用到的時候在仔細研究下）
/**
* A filter, based on the ColumnCountGetFilter, takes two arguments: limit and offset.
* This filter can be used for row-based indexing, where references to other tables are stored across many columns,
* in order to efficient lookups and paginated results for end users.
*/
Filter filter = new ColumnPaginationFilter(5, 15);
Scan scan = new Scan();
scan.setFilter(filter);
ResultScanner scanner = table.getScanner(scan);
// ^^ ColumnPaginationFilterExample
System.out.println("Results of scan:");
// vv ColumnPaginationFilterExample
for (Result result : scanner) {
System.out.println(result);
}
scanner.close(); 2.11 ColumnPrefixFilter
// 跟prefxiFilter相似，只是改成了Column，例項如下：
// vv ColumnPaginationFilterExample
Filter filter = new ColumnPrefixFilter(Bytes.toBytes("qual2"));
Scan scan = new Scan();
scan.setFilter(filter);
ResultScanner scanner = table.getScanner(scan);
// ^^ ColumnPaginationFilterExample
System.out.println("Results of scan:");
// vv ColumnPaginationFilterExample
for (Result result : scanner) {
System.out.println(result);
}
scanner.close();
// 值scan到與列值與前面匹配的資料。例如qual2匹配qual21。2.12 RandomRowFilter
// 隨即的返回row的資料，建構函式為
RandomRowFilter(float chance)
// chance取值為0到1.0，如果<0則為空，如果>1則包含所有的行。3. Decorating Filters
3.1 SkipFilter
//這個過濾器只作用到keyValueFilter上。KeyValueFilter會返回所有滿足條件的row及對應的列。
而加上SkipFilter以後。會發現如果某一行的某一列不符合條件，則這一行全部不返回了。
public static void main(String[] args) throws IOException {
Configuration conf = HBaseConfiguration.create();
HBaseHelper helper = HBaseHelper.getHelper(conf);
helper.dropTable("testtable");
helper.createTable("testtable", "colfam1");
System.out.println("Adding rows to table...");
helper.fillTable("testtable", 1, 30, 5, 2, true, true, "colfam1");
HTable table = new HTable(conf, "testtable");
// vv SkipFilterExample
Filter filter1 = new ValueFilter(CompareFilter.CompareOp.NOT_EQUAL,
new BinaryComparator(Bytes.toBytes("val-0")));
Scan scan = new Scan();
scan.setFilter(filter1); // co SkipFilterExample-1-AddFilter1 Only add the ValueFilter to the first scan.
ResultScanner scanner1 = table.getScanner(scan);
// ^^ SkipFilterExample
System.out.println("Results of scan #1:");
int n = 0;
// vv SkipFilterExample
for (Result result : scanner1) {
for (KeyValue kv : result.raw()) {
System.out.println("KV: " + kv + ", Value: " +
Bytes.toString(kv.getValue()));
// ^^ SkipFilterExample
n++;
// vv SkipFilterExample
}
}
scanner1.close();
Filter filter2 = new SkipFilter(filter1);
scan.setFilter(filter2); // co SkipFilterExample-2-AddFilter2 Add the decorating skip filter for the second scan.
ResultScanner scanner2 = table.getScanner(scan);
// ^^ SkipFilterExample
System.out.println("Total KeyValue count for scan #1: " + n);
n = 0;
System.out.println("Results of scan #2:");
// vv SkipFilterExample
for (Result result : scanner2) {
for (KeyValue kv : result.raw()) {
System.out.println("KV: " + kv + ", Value: " +
Bytes.toString(kv.getValue()));
// ^^ SkipFilterExample
n++;
// vv SkipFilterExample
}
}
scanner2.close();
// ^^ SkipFilterExample
System.out.println("Total KeyValue count for scan #2: " + n);
}3.2 WhileMatchFilters
相當於while執行，知道不match就break了返回了。