1. 程式人生 > >使用 FileSystem JAVA API 對 HDFS 進行讀、寫、刪除等操作

使用 FileSystem JAVA API 對 HDFS 進行讀、寫、刪除等操作

Below is a code sample of how to read from and write to HDFS in java. 

1. Creating a configuration object:  To be able to read from or write to HDFS, you need to create a Configuration object and pass configuration parameter to it using hadoop configuration files.  
  
    // Conf object will read the HDFS configuration parameters from these
  XML
    // files. You may specify the parameters for your own if you want.
 

    Configuration conf = new Configuration(); 
    conf.addResource(new Path("/opt/hadoop-0.20.0/conf/core-site.xml")); 
    conf.addResource(new Path("/opt/hadoop-0.20.0/conf/hdfs-site.xml")); 

    If you do not assign the configurations to conf object (using hadoop xml file) your HDFS operation will be performed on the local file system and not on the HDFS. 

2. Adding file to HDFS:
 Create a FileSystem object and use a file stream to add a file. 

    FileSystem fileSystem = FileSystem.get(conf);
    
    // Check if the file already exists

    Path path = new Path("/path/to/file.ext");
    if (fileSystem.exists(path)) {
        System.out.println("File " + dest + " already exists");

        return;
    }

    // Create a new file and write data to it.
    FSDataOutputStream out = fileSystem.create(path);
    InputStream in = new BufferedInputStream(new FileInputStream(
        new File(source)));


    byte[] b = new byte[1024];
    int numBytes = 0;
    while ((numBytes = in.read(b)) > 0) {
        out.write(b, 0, numBytes);
    }

    // Close all the file descripters
    in.close();
    out.close();
    fileSystem.close();

3. Reading file from HDFS: Create a file stream object to a file in HDFS and read it. 

    FileSystem fileSystem = FileSystem.get(conf);

    Path path = new Path("/path/to/file.ext");
 
    if (!fileSystem.exists(path)) { 
        System.out.println("File does not exists"); 
        return; 
    }

    FSDataInputStream in = fileSystem.open(path);
 

    String filename = file.substring(file.lastIndexOf('/') + 1,
        file.length());
 

    OutputStream out = new BufferedOutputStream(new FileOutputStream(
        new File(filename)));
 

    byte[] b = new byte[1024]; 
    int numBytes = 0; 
    while ((numBytes = in.read(b)) > 0) { 
        out.write(b, 0, numBytes); 
    } 

    in.close(); 
    out.close(); 
    fileSystem.close(); 

3. Deleting file from HDFS: Create a file stream object to a file in HDFS and delete it. 

    FileSystem fileSystem = FileSystem.get(conf); 

    Path path = new Path("/path/to/file.ext"); 
    if (!fileSystem.exists(path)) { 
        System.out.println("File does not exists"); 
        return; 
    }

    // Delete file
    fileSystem.delete(new Path(file), true);
 

    fileSystem.close(); 

3. Create dir in HDFS: Create a file stream object to a file in HDFS and read it. 

    FileSystem fileSystem = FileSystem.get(conf); 

    Path path = new Path(dir); 
    if (fileSystem.exists(path)) { 
        System.out.println("Dir " + dir + " already not exists"); 
        return; 
    }

    // Create directories
    fileSystem.mkdirs(path);
 

    fileSystem.close(); 

Code: