File assets

There is two types of assets. Files that are uploaded and URLs. We will cover Uploading file assets here, for URL assets view this page.

Uploading files

When uploading files they go directly to S3 (or service that implements the S3 API, eg. Minio, DigitalOcean Spaces, Linode Object Storage, etc...)

Firstly we need to send the metadata about the file to the app. In the response will be signed URLs to allow sending the file in chunks directly to S3. Once the file has been uploaded we will need to notify the app that the file has completed and pass back the id for each chunk that was uploaded.

Step 1. Sending the metadata

Send a POST request with metadata.

POST https://app.digital-downloads.com/api/v1/assets/signed

{
    "name": "MyAmazingFile.zip",
    "size": 303287472, // filesize in bytes
    "mime": "application/zip"
}

Response

{
    "chunk_size": 100000000,
    "upload_id": "56f30001-2548-489b-be0b-068be4b8102d",
    "urls": [
        {
            "start": 0,
            "end": 100000000,
            "part": 1,
            "url": "https://minio.massivemonkey.io/digital-assets/my-store.myshopify.com/8b0228e8-4f54-4beb-ae84-da8cc149bc60?uploadId=56f30001-2548-489b-be0b-068be4b8102d&partNumber=1&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=massivemonkey%2F20220629%2Fnyc3%2Fs3%2Faws4_request&X-Amz-Date=20220629T015644Z&X-Amz-SignedHeaders=host&X-Amz-Expires=43200&X-Amz-Signature=5145422d9dcfef6374eb4939619735505c3c6e2a171945f917f74294812ec6e3"
        },
        {
            "start": 100000000,
            "end": 200000000,
            "part": 2,
            "url": "https://minio.massivemonkey.io/digital-assets/my-store.myshopify.com/8b0228e8-4f54-4beb-ae84-da8cc149bc60?uploadId=56f30001-2548-489b-be0b-068be4b8102d&partNumber=2&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=massivemonkey%2F20220629%2Fnyc3%2Fs3%2Faws4_request&X-Amz-Date=20220629T015644Z&X-Amz-SignedHeaders=host&X-Amz-Expires=43200&X-Amz-Signature=4f56dbbebc3002a8df0037f5a9192d00490179e23af4f0e72894dd61e93c87ea"
        },
        {
            "start": 200000000,
            "end": 300000000,
            "part": 3,
            "url": "https://minio.massivemonkey.io/digital-assets/my-store.myshopify.com/8b0228e8-4f54-4beb-ae84-da8cc149bc60?uploadId=56f30001-2548-489b-be0b-068be4b8102d&partNumber=3&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=massivemonkey%2F20220629%2Fnyc3%2Fs3%2Faws4_request&X-Amz-Date=20220629T015644Z&X-Amz-SignedHeaders=host&X-Amz-Expires=43200&X-Amz-Signature=447af2a5eef971882a2028fcfd6c1945b37b8c76dd2edd4d3be2def08abe1447"
        },
        {
            "start": 300000000,
            "end": 303287472,
            "part": 4,
            "url": "https://minio.massivemonkey.io/digital-assets/my-store.myshopify.com/8b0228e8-4f54-4beb-ae84-da8cc149bc60?uploadId=56f30001-2548-489b-be0b-068be4b8102d&partNumber=4&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=massivemonkey%2F20220629%2Fnyc3%2Fs3%2Faws4_request&X-Amz-Date=20220629T015644Z&X-Amz-SignedHeaders=host&X-Amz-Expires=43200&X-Amz-Signature=485b0bee23986b5add37dfaf0422a95e897949789f0ec64c0c880237f39ccf2b"
        }
    ],
    "id": "e784c8cb-7192-4c6c-9004-17fe31a859fc",
    "file_url": "https://app.digital-downloads.com/assets/e784c8cb-7192-4c6c-9004-17fe31a859fc"
}

The urls that are generated are chunked into 100MB parts (but this could change at any time, the chunk size is provided in the payload). Using the URLs send a PUT request to each with the section of the file using the start and end bytes. These chunks do not need to be uploaded in order or wait until the previous part has finished.

Step 2. Uploading file parts to S3

Send a PUT request with chunk of the file. See JavaScript example.

for (let part of urls) {
    // get only the part of the file that is required
    let blob = file.slice(part.start, part.end);


    // create a new http request and remove the content type
    const a = axios.create();
    delete a.defaults.headers.put['Content-Type'];


    a.put(part.url, blob).then((response) => {
        // from the header we need to etag, this is S3's id for each part
        // of file so we can re construct it when all pieces are uploaded
        return {
            ETag: response.headers.etag.split('"').join(''), 
            PartNumber: part.part,
        };
    });
}

Step 3. Notifying the app that the file has uploaded

Once each chunk has been uploaded to S3 send a POST request with each chunk details back to the app.

POST https://app.digital-downloads.com/api/v1/assets/:id/uploaded

{
   "parts":[
      {
         "ETag": "0d5bf6a8d41ed4d3d8813fa27ee67b6f",
         "PartNumber": 1
      },
      {
         "ETag": "4f0d43ac4cd8dc4ec545572c44b8642b",
         "PartNumber": 2
      },
      {
         "ETag": "00dbc7cf67bef6ae9eb5c47936f1f289",
         "PartNumber": 3
      },
      {
         "ETag": "dd2b66242dba17ac0b7e33b7a70e716b",
         "PartNumber": 4
      }
   ],
   "upload_id": "56f30001-2548-489b-be0b-068be4b8102d"
}

JavaScript example of the flow

This is an example of the code used within the UI of the app to upload files.

async function upload(file) {
    // generate the signed urls
    let signedResponse = await axios.post('https://app.digital-downloads.com/api/v1/assets/signed', {
        name: file.name,
        size: file.size,
        mime: file.mime,
    }).then((r) => r.data);


    // create a new http request and remove the content type
    const a = axios.create();
    delete a.defaults.headers.put['Content-Type'];


    const promises = [];


    for (let part of signedResponse.urls) {
        // get the part of the file to send in this request
        let blob = file.file.slice(part.start, part.end);


        promises.push(
            // send the PUT request to the url with part of the file
            a.put(part.url, blob).then((response) => {
                // from the header we need to etag, this is S3's id for each part
                // of file so we can re construct it when all pieces are uploaded
                return {
                    ETag: response.headers.etag.split('"').join(''),
                    PartNumber: part.part,
                };
            })
        );
    }


    // once all parts have uploaded then we need to send 
    // the parts with the joined S3 etag back to the app
    Promise.all(promises).then((parts) => {
        axios.put(`https://app.digital-downloads.com/api/v1/assets/${signedResponse.id}/uploaded`, { 
            parts: parts, 
            upload_id: signedResponse.upload_id 
        }).then((r) => {
            // completed
        });
    });
}

Canceling an upload

If a chunk fails or something goes wrong, or you want to cancel the upload. We need to remove the chunks that got uploaded but are no longer needed. Send a POST request to the asset with the upload id (received from the signed request).

POST https://app.digital-downloads.com/api/v1/assets/:id/cancel

{
    "upload_id": "56f30001-2548-489b-be0b-068be4b8102d"
}