Struct opendal::services::Webhdfs

pub struct Webhdfs { /* private fields */ }

WebHDFS’s REST API support.

There two implementations of WebHDFS REST API:

• Native via HDFS Namenode and Datanode, data are transferred between nodes directly.
• HttpFS is a gateway before hdfs nodes, data are proxied.

Capabilities

This service can be used to:

• read
• write
• list
• scan
• presign
• blocking

Differences with hdfs

Hdfs is powered by HDFS’s native java client. Users need to setup the hdfs services correctly. But webhdfs can access from HTTP API and no extra setup needed.

Configurations

• root: The root path of the WebHDFS service.
• endpoint: The endpoint of the WebHDFS service.
• delegation: The delegation token for WebHDFS.

Refer to Builder’s public API docs for more information

Examples

Via Builder

use std::sync::Arc;

use anyhow::Result;
use opendal::services::Webhdfs;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder = Webhdfs::default();
    // set the root for WebHDFS, all operations will happen under this root
    //
    // Note:
    // if the root is not exists, the builder will automatically create the
    // root directory for you
    // if the root exists and is a directory, the builder will continue working
    // if the root exists and is a folder, the builder will fail on building backend
    builder.root("/path/to/dir");
    // set the endpoint of webhdfs namenode, controlled by dfs.namenode.http-address
    // default is http://127.0.0.1:9870
    builder.endpoint("http://127.0.0.1:9870");
    // set the delegation_token for builder
    builder.delegation("delegation_token");

    let op: Operator = Operator::new(builder)?.finish();

    Ok(())
}

Implementations

impl WebhdfsBuilder

pub fn root(&mut self, root: &str) -> &mut Self

Set the working directory of this backend

All operations will happen under this root

Note

The root will be automatically created if not exists. If the root is occupied by a file, building of directory will fail

pub fn endpoint(&mut self, endpoint: &str) -> &mut Self

Set the remote address of this backend default to http://127.0.0.1:9870

Endpoints should be full uri, e.g.

• https://webhdfs.example.com:9870
• http://192.168.66.88:9870

If user inputs endpoint without scheme, we will prepend http:// to it.

pub fn delegation(&mut self, delegation: &str) -> &mut Self

Set the delegation token of this backend, used for authentication

Note

The builder prefers using delegation token over username. If both are set, delegation token will be used.

Trait Implementations

impl Builder for WebhdfsBuilder

fn build(&mut self) -> Result<Self::Accessor>

build the backend

Note:

when building backend, the built backend will check if the root directory exits. if the directory does not exits, the directory will be automatically created if the root path is occupied by a file, a failure will be returned

const SCHEME: Scheme = Scheme::Webhdfs

Associated scheme for this builder.

type Accessor = WebhdfsBackend

The accessor that built by this builder.

fn from_map(map: HashMap<String, String>) -> Self

Construct a builder from given map.

fn from_iter(iter: impl Iterator<Item = (String, String)>) -> Selfwhere
    Self: Sized,

Construct a builder from given iterator.

fn from_env() -> Selfwhere
    Self: Sized,

Construct a builder from envs.

impl Clone for WebhdfsBuilder

fn clone(&self) -> WebhdfsBuilder

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for WebhdfsBuilder

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for WebhdfsBuilder

fn default() -> WebhdfsBuilder

Returns the “default value” for a type.