"fs.s3.awsAccessKeyId", access_key
"fs.s3n.awsAccessKeyId", access_key
"fs.s3a.access.key", access_key
"fs.s3.awsSecretAccessKey", secret_key
"fs.s3n.awsSecretAccessKey", secret_key
"fs.s3a.secret.key", secret_key
"fs.s3n.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem"
"fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem"
"fs.s3.impl", "org.apache.hadoop.fs.s3.S3FileSystem"
If one needs to copy data from one S3 Bucket to other with different credential keys. Then -
- If you are on Hadoop cluster with version 2.7, and using s3a:// then -
- use URI as following - s3a://DataAccountKey:DataAccountSecretKey/DataAccount/path
- If you are on EMR or Hadoop 2.8+ then one can add properties per-bucket, as following -
- fs.s3a.bucket.DataAccount.access.key DataAccountKey fs.s3a.bucket.DataAccount.secret.key DataAccountSecretKey
- fs.s3.bucket.DataAccount.awsAccessKeyId DataAccountKey fs.s3.bucket.DataAccount.awsSecretAccessKey DataAccountSecretKey
- fs.s3n.bucket.DataAccount.awsAccessKeyId DataAccountKey fs.s3n.bucket.DataAccount.awsSecretAccessKey DataAccountSecretKey
Note - If you are using Spark then prepend spark.hadoop. to above properties.
Comments
Post a Comment