Skip to content

Update 2023-07-25-announcing-cpp.md #1433

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Aug 16, 2023
Merged

Update 2023-07-25-announcing-cpp.md #1433

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cb23617
Update 2023-07-25-announcing-cpp.md
johnhe-dev Aug 15, 2023
16fd958
Merge pull request #1 from johnhe-dev/johnhe-dev-patch-1
johnhe-dev Aug 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions _posts/2023-07-25-announcing-cpp.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
layout: blog_detail
title: "Announcing CPP-based S3 IO DataPipes"
author: John He, Khaled ElGalaind Roshani Nagmote, Daiming Yang
author: John He, Khaled ElGalaind, Roshani Nagmote, Daiming Yang
---

Training large deep learning models requires large datasets. [Amazon Simple Storage Service](https://aws.amazon.com/s3/) (Amazon S3) is a scalable cloud object store service used for storing large training datasets. Machine learning (ML) practitioners need an efficient data pipe that can download data from Amazon S3, transform the data, and feed the data to GPUs for training models with high throughput and low latency.
Expand Down Expand Up @@ -48,7 +48,7 @@ The following code snippet provides a typical usage of `load_files_by_s3()`:
from torch.utils.data import DataLoader

from torchdata.datapipes.iter import IterableWrapper



s3_shard_urls = IterableWrapper(["s3://bucket/prefix/",])

s3_shard_urls = IterableWrapper(["s3://bucket/prefix/",])
.list_files_by_s3()
s3_shards = s3_shard_urls.load_files_by_s3()

# text data

training_data = s3_shards.readlines(return_path=False)

Expand Down