Configure an externally managed bucket

Bobsled can deliver data to an external bucket, housed within a non-Bobsled AWS account. For example, this could be a bucket in the provider's AWS account or in a consumer's AWS account. This provides maximum flexibility and control over the bucket itself.

Setting up a share to an external bucket in Bobsled

1. On the share page, click the box Choose Destination

2. Choose the cloud platform Amazon S3 and choose the region of the target bucket

3. Select "External bucket" and press continue

Set up destination access

Bobsled has flexible options for how to write to an external bucket

Provide the bucket name and optional path to write to:

If desired you can enable

• Bobsled Share Path to add "share ID" and "latest" to the path written by Bobsled. This should be used when delivering via multiple shares to the same bucket to ensure they don’t overlap

• Mirror source data to allow Bobsled to delete files that are removed from the source. This mode tells Bobsled to match the contents of the source bucket to the destination. When using a DW source or when removing files is not required, this setting is suggested to be off.

Finally, select how you'd like Bobsled to access the bucket:

• Assume role - you give a Bobsled ARN the ability to assume a role within the external account

• Access key - you provide an AWS access key and secret

• Consumer role - you give a Bobsled ARN the ability to assume a consumer role within the external account

"Assume role" setup

Prerequisites 

To configure Bobsled access to the bucket, your user must have the sufficient permissions to create policies and assign roles in Amazon Web Services (AWS).

(1) Create an IAM Policy

1. Login to AWS Management Console

2. From the Services dropdown, select IAM under Security, Identify & Compliance section

3. Click Account Settings on the left-hand panel

4. Expand the Security Token Service (STS) list, find the AWS region corresponding to the region where your bucket is located, and choose Activate if the status is Inactive.

5. Choose Policies from the left-hand navigation pane

6. Click Create Policy

7. Click the JSON tab

8. Add JSON policy that allows Bobsled to write to the S3 bucket. The following policies provide Bobsled with required permissions to write data to the bucket or a specific path. Please replace placeholders with your bucket name(s), and pay special attention to the trailing /*. It should be present on the first statement, but not on the second one. 

    8a. Allow Bobsled to write to an entire bucket. 

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:GetObjectAttributes",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket-name>/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket-name>"
      ]
    }
  ]
}

    8b. Allow Bobsled to write to subfolders within a bucket. 

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:GetObjectAttributes",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket-name>/subfolder/*"
      ]
    },
    {
      "Sid": "AllowListingOfSubFolder",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::<bucket-name>"
      ],
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "subfolder/*"
          ]
        }
      }
    }
  ]
}

9. Create a policy name (e.g. bobsled_access) and an optional description. Click Create policy. 

(2) Create an IAM Role

1. Login to AWS Management Console 

2. From the Services dropdown, select IAM under the Security, Identity & Compliance header 

3. Click Roles on the left-hand panel

4. Click the Create role button. 

5. Under Trusted entity type, select Custom trust policy

6. Set the trust policy using the following json:

    6a. Replace the <awsBobsledWireArn> & <awsBobsledWriteExternalId> in the json below with the values found in the destination setup flow within Bobsled.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "<awsBobsledWriteArn>"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<awsBobsledWriteExternalId>"
        }
      }
    }
  ]
}

7. Click the Next button 

8. Find the policy you created in the previous section, and select this policy 

9. Click the Next button 

10. Enter a name and description for the role, and click the Create role button 

11. Record the Role ARN value located on the role summary page. You will use the Role ARN to configure your destination bucket in Bobsled.

For KMS Encrypted Buckets Only

(3) Grant Bobsled Role Access to Encryption Keys

1. Navigate to the S3 bucket, click on Properties > "Default encryption" section. 

2. Click the link for the Encryption key ARN

3. Scroll down to the "Key users" section. Click Add. 

4. Search for the role created for Bobsled in the previous section. Select the checkbox to the left of the role. Click Add.

"Consumer role" setup

Prerequisites 

To configure Bobsled access to the bucket, your user must have the sufficient permissions to create policies and assign roles in Amazon Web Services (AWS). 

(1) Create an IAM Policy (Provider)

1. Login to AWS Management Console

2. From the Services dropdown, select IAM under Security, Identify & Compliance section

3. Click Account Settings on the left-hand panel

4. Expand the Security Token Service (STS) list, find the AWS region corresponding to the region where your bucket is located, and choose Activate if the status is Inactive.

5. Choose Policies from the left-hand navigation pane

6. Click Create Policy

7. Click the JSON tab

8. Add JSON policy that allows Bobsled to assume the role. The following policies provide Bobsled with the required permissions to assume a role that will be used to assume a Consumer role. 

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "AllowAssumingRoleInConsumerAWS",
			"Effect": "Allow",
			"Action": [
				"sts:GetSessionToken",
				"sts:AssumeRole",
				"sts:GetCallerIdentity"
			],
			"Resource": "*"
		}
	]
}

9. Create a policy name (e.g. bobsled_access) and an optional description. Click Create policy. 

(2) Create an IAM Role (Provider)

1. Login to AWS Management Console 

2. From the Services dropdown, select IAM under the Security, Identity & Compliance header 

3. Click Roles on the left-hand panel

4. Click the Create role button. 

5. Under Trusted entity type, select Custom trust policy

6. Set the trust policy using the following json:

    6a. Replace the <awsBobsledWriteArn> & <awsBobsledWriteExternalId> in the json below with the values found in the destination setup flow within Bobsled. You can skip 

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "<awsBobsledWriteArn>"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<awsBobsledWriteExternalId>"
        }
      }
    }
  ]
}

7. Click the Next button 

8. Find the policy you created in the previous section, and select this policy 

9. Click the Next button 

10. Enter a name and description for the role, and click the Create role button 

11. Record the Role ARN value located on the role summary page. You will use the Role ARN to configure your destination bucket in Bobsled.

(3) Create an IAM Policy (Consumer)

1. Login to AWS Management Console

2. From the Services dropdown, select IAM under Security, Identify & Compliance section

3. Click Account Settings on the left-hand panel

4. Expand the Security Token Service (STS) list, find the AWS region corresponding to the region where your bucket is located, and choose Activate if the status is Inactive.

5. Choose Policies from the left-hand navigation pane

6. Click Create Policy

7. Click the JSON tab

8. Add JSON policy that allows Bobsled to write to the S3 bucket. The following policies provide Bobsled with required permissions to write data to the bucket or a specific path. Please replace placeholders with your bucket name(s), and pay special attention to the trailing /*. It should be present on the first statement, but not on the second one. 

    8a. Allow Bobsled to write to an entire bucket. 

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:GetObjectAttributes",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket-name>/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket-name>"
      ]
    }
  ]
}

    8b. Allow Bobsled to write to subfolders within a bucket. 

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion",
        "s3:GetObjectAttributes",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket-name>/subfolder/*"
      ]
    },
    {
      "Sid": "AllowListingOfSubFolder",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::<bucket-name>"
      ],
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "subfolder/*"
          ]
        }
      }
    }
  ]
}

9. Create a policy name (e.g. bobsled_access) and an optional description. Click Create policy. 

(4) Create an IAM Role (Consumer)

1. Login to AWS Management Console 

2. From the Services dropdown, select IAM under the Security, Identity & Compliance header 

3. Click Roles on the left-hand panel

4. Click the Create role button. 

5. Under Trusted entity type, select Custom trust policy

6. Set the trust policy using the following json:

    6a. Replace the <awsProviderWireArn> in the json below with the value provided by the provider. You can skip "Condition" part of the policy, but if you choose to set it - let the provider know

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "<awsProviderdWriteArn>"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<awsProviderWriteExternalId>"
        }
      }
    }
  ]
}

7. Click the Next button 

8. Find the policy you created in the previous section, and select this policy 

9. Click the Next button 

10. Enter a name and description for the role, and click the Create role button 

11. Record the Role ARN value located on the role summary page. You will use the Role ARN to configure your destination bucket in Bobsled.