S3 event notifications
When you use your own S3 storage, files can also be added, removed or moved to another storage class directly in the bucket, outside MediaLab. With S3 event notifications, your bucket reports these changes to an Amazon SQS queue, and MediaLab reads that queue to keep your account in sync.
How MediaLab keeps up with your bucket
Each event tells MediaLab which object changed. MediaLab then looks at the object that is in the bucket at that moment and compares it with what it knows. It doesn't rely on the type of event alone, because events can arrive late or out of order, and an overwrite is only reported as an upload. Only the current version of an object counts.
An object counts as different when its size changed, or its checksum changed where both sides know one.
What happens in each case
| What happens in your bucket | What MediaLab does |
|---|---|
| A new object is uploaded | A new file is added, in the folder that matches the object's path. Missing folders are created. |
| An object is copied to a new key | Same as a new upload. |
| An object is overwritten with different content | The existing file is moved to the trash, without source. A new file is added for the new content. |
| An object is overwritten with the same content | Nothing changes. |
| An object is copied onto itself with another storage class | The storage type is updated. See Storage tiering. |
| An object is deleted | Files that used the object as their source are moved to the trash, without source. Formats stored as that object are no longer available. |
| An object is deleted and uploaded again before MediaLab processed the delete | The old file is moved to the trash and the new upload is added as a new file. |
| A lifecycle rule expires an object | Same as a delete. |
| A lifecycle rule or Intelligent-Tiering moves an object to another storage class | The storage type is updated, and the file is marked archived (or no longer archived) to match. |
| MediaLab itself uploads or deletes an object | Nothing extra happens: MediaLab already knows about the change. |
Buckets with versioning
| What happens in your bucket | What MediaLab does |
|---|---|
| An object is deleted, which adds a delete marker | Same as a delete: the file is moved to the trash, without source. |
| A lifecycle rule expires the current version, which adds a delete marker | Same as a delete. |
| A new version of an object is uploaded | Same as an overwrite: with different content, the existing file goes to the trash and a new file is added. |
| A delete marker is removed, so the object is back | The object is added as a new file. The file that was moved to the trash earlier stays there, without source. |
| The current version is permanently deleted, so the previous version is current again | Handled as an overwrite: the existing file goes to the trash, and the previous version is added as a new file. |
| A noncurrent (older) version is permanently deleted | Nothing changes: the current version is still the same. |
MediaLab always uses the current version of an object. A file in the trash doesn't keep a link to an older version of its object.
Good to know
- Only paths inside a storage container linked to the queue are processed. Changes elsewhere in the bucket are ignored.
- A file moved to the trash because its object was deleted or replaced can still be found in the trash, but it has no source anymore: its original can't be downloaded or used to make new formats, also not after restoring it. Emptying the trash removes it for good.
- Locked files are not moved to the trash. They stay in place, without source.
- If a file's source is also stored in MediaLab's own storage, removing the object from your bucket only removes that cloud copy. The file keeps working.
- An overwrite with exactly the same size can only be recognized when a checksum is known. That isn't the case for large (multipart) uploads or objects encrypted with your own KMS key, so such an overwrite keeps the existing file.
- If MediaLab can't reach your bucket while handling an event, nothing changes for that event.
Setting it up in AWS
You need access to the AWS console for the account that owns the bucket.
1. Create the queue
- Open Amazon SQS in the same region as your bucket and click Create queue.
- Choose type Standard. FIFO queues can't receive S3 event notifications.
- Give the queue a name, for example
medialab-events. - Under Access policy, choose Advanced and allow your bucket to send messages to the queue:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Service": "s3.amazonaws.com" },
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:<region>:<account-id>:<queue-name>",
"Condition": {
"ArnEquals": { "aws:SourceArn": "arn:aws:s3:::<bucket-name>" },
"StringEquals": { "aws:SourceAccount": "<account-id>" }
}
}
]
}
- If you enable encryption with your own KMS key (SSE-KMS), the key policy must also allow
s3.amazonaws.comto use the key. Encryption with SQS-managed keys (SSE-SQS) needs nothing extra. - Create the queue.
2. Send the bucket's events to the queue
- Open your bucket in Amazon S3 and go to Properties → Event notifications → Create event notification.
- Optionally set a prefix to only send events for the path MediaLab uses.
- Select these event types:
- All object create events (
s3:ObjectCreated:*) - All object removal events (
s3:ObjectRemoved:*), which includes delete markers - Lifecycle expiration (
s3:LifecycleExpiration:*), if a lifecycle rule expires objects - Lifecycle transition (
s3:LifecycleTransition), if a lifecycle rule moves objects to another storage class - Intelligent-Tiering archive (
s3:IntelligentTiering), if you use Intelligent-Tiering archive access tiers
Other event types are ignored. - All object create events (
- Under Destination, choose SQS queue and select the queue you created.
- Save. S3 sends a test message to the queue to check the setup. MediaLab ignores it.
3. Give MediaLab access to the queue
MediaLab reads the queue with the same credentials it uses for your bucket (the access key, or the role MediaLab assumes). Add this statement to that user's or role's IAM policy:
{
"Effect": "Allow",
"Action": [
"sqs:ReceiveMessage",
"sqs:DeleteMessage"
],
"Resource": "arn:aws:sqs:<region>:<account-id>:<queue-name>"
}
MediaLab removes each message from the queue once it has been processed.
Connecting the queue to MediaLab
Send the following to MediaLab's service desk, and say which storage containers the queue is for:
- the queue name;
- the region of the queue;
- the AWS account ID that owns the queue.
Once the queue is connected, changes in the bucket show up in MediaLab shortly after they happen.
Troubleshooting
- Nothing happens after an upload: check that the object's path is inside a storage container linked to the queue, that the event types above are selected, and that the queue's Monitoring tab shows messages arriving.
- Messages stay in the queue: MediaLab can't read the queue. Check the IAM permissions from step 3, and that the queue name, region and account ID are correct.
- Deleted files stay in MediaLab: check that All object removal events is selected, so delete markers in a versioned bucket are reported too, and Lifecycle expiration if a lifecycle rule removes the objects.
- A file was replaced by a new file: the object at that key was overwritten with different content, or the current version was deleted in a versioned bucket. The previous file is in the trash.