I published an article recently about pulling 2 TB out of S3 Glacier Deep Archive through CloudFront for 15 dollars flat rate instead of 200. The article ended before the story did. There was one more catch, and this is that part.

194 objects in total. 45 to 50 GB was most of them. Six of them 250 to 350 GB, old backups I had never split. I kicked off the download and went to bed.

Six files, stuck on failed

By evening I came back and found six files marked failed. Everything else had gone through fine. My first thought was that I had been blocked somewhere, too many parallel connections, WAF, some rate limit. I went through the logs. Nothing that looked like it.

The browser was more honest than the logs. Direct link to one of the six files, and this is what came back.

ERROR: The request could not be satisfied.
ERROR: The request could not be satisfied.

No status code that explained anything. No mention of size. Just a request ID and a timestamp.

The limit I didn't know about

And turns out CloudFront has a hard limit I had no idea existed. From the official quota page, under "General quotas on distributions".

Maximum cacheable file size per HTTP GET response: 50 GB

My six files were way past that, one of them close to seven times over. And looking back, it explains why the 45-to-50 GB objects made it through at all. They had been sitting right at the wall the whole time, and I never noticed.

For a moment I was ready to just pay for the extra 500 GB of egress, to hell with it, and pull the six files straight from S3, around the whole CloudFront quota. That would have undone the entire point of the first article, but I don't like giving up.

Why chunks work

I asked an AI to look into it, and what came back was something I hadn't known either. This isn't a workaround, it's documented by AWS itself. From the range GET guide:

When caching is enabled, CloudFront doesn't retrieve or cache an object that is larger than 50 GB (...) However, with range requests, you can use CloudFront to cache an object that is larger than the maximum cacheable file size.

AWS documentation:

Not a hack. The API is exactly working as designed, you just have to know you're supposed to use it this way.

CloudFront doesn't split a large file on its own. Instead the client has to do that itself, through the Range header, each chunk under 50 GB, and CloudFront caches and serves every part on its own.

The script

So, one curl per chunk, with Range: bytes=START-END, 40 GB per part to stay comfortably under the limit. With a Range: bytes=0-0 request and the response's Content-Range header, the script fetches the total size itself. Each part lands in its own .part.NNNN file, and before every download the script checks whether that file already has the expected size. If not, it fetches it again. That makes the whole run resumable, no re-downloading parts that already finished.

URL="PATH_TO_LARGE_CLOUDFRONT_FILE"
OUT="$HOME/Downloads/NAME_OF_THE_CLOUDFRONT_FILE"
CHUNK=$((40*1024*1024*1024)) # 40 GB per chunk, with margin under the 50 GB limit

TOTAL=$(curl -sD - -o /dev/null -A "Mozilla/5.0" -H "Range: bytes=0-0" "$URL" \
  | awk -F/ 'tolower($0) ~ /content-range:/ {gsub("\r","",$2); print $2; exit}')
echo "size: $TOTAL"
[ -n "$TOTAL" ] || exit 1

START=0
PART=0
while [ "$START" -lt "$TOTAL" ]; do
  END=$((START + CHUNK - 1))
  [ "$END" -ge "$TOTAL" ] && END=$((TOTAL - 1))
  PART_FILE=$(printf "%s.part.%04d" "$OUT" "$PART")
  NEED=$((END - START + 1))
  HAVE=0
  [ -f "$PART_FILE" ] && HAVE=$(wc -c < "$PART_FILE" | tr -d ' ')
  if [ "$HAVE" -ne "$NEED" ]; then
    echo "part $PART  $START-$END"
    curl --fail --retry 20 --retry-all-errors --retry-delay 5 \
      -A "Mozilla/5.0" \
      -H "Range: bytes=${START}-${END}" \
      -o "$PART_FILE" \
      "$URL" || exit 1
  else
    echo "part $PART already ok"
  fi
  START=$((END + 1))
  PART=$((PART + 1))
done

cat "$OUT".part.* > "$OUT" && rm -f "$OUT".part.*
ls -lh "$OUT"
# You'll see the 40 GB parts only briefly. After the cat it's one file again.

A word on --retry-all-errors. Without it, curl doesn't abort on some error codes, it just writes out an empty or truncated file, and that's exactly what the HAVE-versus-NEED check on the next run catches.

What it cost

Fifteen dollars is still fifteen dollars. Under 4% of the 50 TB CloudFront quota used. I could have paid the full two hundred for the same result, if I had given up on the first attempt.

Everything now lives on the NAS and, on top of that, on Proton Drive (Visionary, 6TB). In theory I could turn AWS off entirely now. In practice not yet, not until I buy another HDD and copy everything over for a proper 3-2-1 backup. Storage prices are climbing like crazy right now, HDDs included, I'm almost out of words for it. But hope dies last.

This is just my own experience, might be useful to someone. If you ever hit this same wall, write to me.