How to access Azure datalake using the webhdfs API

Question

I have a data lake and using webHDFS I was not able to access my files which I have uploaded into data lake. It is throwing an error. Can anyone help me with this?

Answer 1

You can do it using curl, below is the code how you can able to do it:

$client_id = "<client id>";

$client_secret = "<secret>";

$tenant = "<tenant>";

$adlsAccount = "<account>";

cd D:\path\to\curl

#authenticate

$cmd = { .\curl.exe -X POST https://login.microsoftonline.com/$tenant/oauth2/token  -F grant_type=client_credentials       -F resource=https://management.core.windows.net/       -F client_id=$client_id       -F client_secret=$client_secret };

$responseToken = Invoke-Command -scriptblock $cmd;

$accessToken = (ConvertFrom-Json $responseToken).access_token;

#list root folders

$cmd = {.\curl.exe -X GET -H "Authorization: Bearer $accessToken" https://$adlsAccount.azuredatalakestore.net/webhdfs/v1/?op=LISTSTATUS };

$foldersResponse = Invoke-Command -scriptblock $cmd;

#loop through directories directories

(ConvertFrom-Json $foldersResponse).FileStatuses.FileStatus | ForEach-Object { $_.pathSuffix }

#list files in one folder

$cmd = {.\curl.exe -X GET -H "Authorization: Bearer $accessToken" https://$adlsAccount.azuredatalakestore.net/webhdfs/v1/weather/?op=LISTSTATUS };

$weatherResponse = Invoke-Command -scriptblock $cmd;

(ConvertFrom-Json $weatherResponse).FileStatuses.FileStatus | ForEach-Object { $_.pathSuffix }

#download one file

$cmd = {.\curl.exe -L "https://$adlsAccount.azuredatalakestore.net/webhdfs/v1/weather/2007small.csv?op=OPEN" -H "Authorization: Bearer $accessToken" -o d:\temp\curl\2007small.csv };

Invoke-Command -scriptblock $cmd;

#upload one file

$cmd = {.\curl.exe -i -X PUT -L "https://$adlsAccount.azuredatalakestore.net/webhdfs/v1/weather/new2007small.csv?op=CREATE" -T "D:\temp\weather\smallcsv\new2007small.csv" -H "Authorization: Bearer $accessToken" };

Invoke-Command -scriptblock $cmd;

I hope this will work.

Want to become an Azure expert? join azure architect certification now!!