|
From: | Michael A Nesmith |
Subject: | Using parallel to move files from EC2 to AWS S3 bucket |
Date: | Wed, 28 Nov 2018 14:08:15 +0000 |
Hi,
I am trying to move files from an EC2 Linux instance to a S3 staging bucket.
I have the parallel option working as follows:
--------------------------------------------------------------------------------------------------------
=====================================================================================================================================
[root@ixe-1a-nfs-10618 predev]# cat mytest3.sh
TRANSACTIONID=ec9ba83b
VERTICALID=rfx
VERSION=predev
FILEPATH=/nfsshare/predev/rfx/inbound
FILENAME=NFS.ICPRIV.R.RF3.AHC.EBB21.1636227-EB_B21_QQ_4.txt
file="/nfsshare/predev/rfx/inbound/NFS.ICPRIV.R.RF3.AHC.EBB21.1636227-EB_B21_QQ_4.txt"
ls "$file" | parallel --will-cite aws s3 mv {} s3://hhs-cms-epl-epor-stg-"${VERSION}"-v3/"$VERTICALID""_""$FILENAME" \
--content-disposition "attachment" \
&& \
echo "file moved to s3 staging bucket"
======================================================================================================================================
[root@ixe-1a-nfs-10618 predev]# ./mytest3.sh
move: ../../../../nfsshare/predev/rfx/inbound/NFS.ICPRIV.R.RF3.AHC.EBB21.1636227-EB_B21_QQ_4.txt to s3://hhs-cms-epl-epor-stg-predev-v3/rfx_NFS.ICPRIV.R.RF3.AHC.EBB21.1636227-EB_B21_QQ_4.txt
file moved to s3 staging bucket
[root@ixe-1a-nfs-10618 predev]# aws s3 ls hhs-cms-epl-epor-stg-predev-v3 | grep NFS.ICPRIV.R.RF3.AHC.EBB21.1636227-EB_B21_QQ_4.txt | wc -l
1
----------------------------------------------------------------------------------------------------------
The problem now seems to be the use of the --metadata option as below: ============================================================================================================================
[root@ixe-1a-nfs-10618 predev]# cat mytest2.sh
TRANSACTIONID=ec9ba83b
VERTICALID=rfx
VERSION=predev
FILEPATH=/nfsshare/predev/rfx/inbound
FILENAME=NFS.ICPRIV.R.RF3.AHC.EBB21.1636227-EB_B21_QQ_4.txt
file="/nfsshare/predev/rfx/inbound/NFS.ICPRIV.R.RF3.AHC.EBB21.1636227-EB_B21_QQ_4.txt"
ls "$file" | parallel --will-cite aws s3 mv {} s3://hhs-cms-epl-epor-stg-"${VERSION}"-v3/"$VERTICALID""_""$FILENAME" \
--content-disposition "attachment" \
--metadata \
"{\"source\":\"nfs\", \
\"originalname\":\"$FILENAME\", \
\"uid\":\"null\", \
\"verticalid\":\"$VERTICALID\", \
\"read\":\"icprivusrgrp\", \
\"write\":\"icprivusrgrp\", \
\"owner\":\"icprivusrgrp\", \
\"transactionID\":\"$TRANSACTIONID\", \
\"callbackurl\":\"null\"}" && \
echo "file moved to s3 staging bucket"
=============================================================================================================================
[root@ixe-1a-nfs-10618 predev]# ./mytest2.sh
Error parsing parameter '--metadata': Invalid JSON: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
JSON received: {source:nfs,
[root@ixe-1a-nfs-10618 predev]#
Any ideas from anyone on how to overcome this? I can now move data to the S3 bucket with the parallel option, but really need to have the metadata set as well.
Thoughts?
Thanks, Mike
|
[Prev in Thread] | Current Thread | [Next in Thread] |