fs: use streaming directory processing in cp() by cjihrig · Pull Request #41351 · nodejs/node

cjihrig · 2021-12-29T19:16:44Z

The readdir() functions do not scale well, which is why opendir(), etc. were introduced. This is exacerbated in the current cp() implementation, which calls readdir() recursively.

This commit updates cp() to use the opendir() style iteration.

The readdir() functions do not scale well, which is why opendir(), etc. were introduced. This is exacerbated in the current cp() implementation, which calls readdir() recursively. This commit updates cp() to use the opendir() style iteration.

bnb · 2021-12-29T19:20:45Z

Would love to see perf metrics on this if you've got any handy 👀

mcollina

lgtm

lib/internal/fs/cp/cp.js

cjihrig · 2021-12-29T19:50:53Z

Would love to see perf metrics on this if you've got any handy

I don't have any. The readdir() based implementation should actually be faster.

Co-authored-by: Luigi Pinca <luigipinca@gmail.com>

nodejs-github-bot · 2021-12-29T21:22:42Z

CI: https://ci.nodejs.org/job/node-test-pull-request/41681/

juanarbol

Awesome!

nodejs-github-bot · 2021-12-31T20:22:57Z

Landed in d0c1176

The readdir() functions do not scale well, which is why opendir(), etc. were introduced. This is exacerbated in the current cp() implementation, which calls readdir() recursively. This commit updates cp() to use the opendir() style iteration. PR-URL: #41351 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Tierney Cyren <hello@bnb.im> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Juan José Arboleda <soyjuanarbol@gmail.com> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>

The readdir() functions do not scale well, which is why opendir(), etc. were introduced. This is exacerbated in the current cp() implementation, which calls readdir() recursively. This commit updates cp() to use the opendir() style iteration. PR-URL: nodejs#41351 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Tierney Cyren <hello@bnb.im> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Juan José Arboleda <soyjuanarbol@gmail.com> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>

The readdir() functions do not scale well, which is why opendir(), etc. were introduced. This is exacerbated in the current cp() implementation, which calls readdir() recursively. This commit updates cp() to use the opendir() style iteration. PR-URL: #41351 Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Tierney Cyren <hello@bnb.im> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Juan José Arboleda <soyjuanarbol@gmail.com> Reviewed-By: Mohammed Keyvanzadeh <mohammadkeyvanzade94@gmail.com>

* refactor(copy): backport nodejs/node#41351 * perf(copy): parallel copy * perf(copy): run filter in parallel as well

fs: use async directory processing in cp()

b5aa0b3

The readdir() functions do not scale well, which is why opendir(), etc. were introduced. This is exacerbated in the current cp() implementation, which calls readdir() recursively. This commit updates cp() to use the opendir() style iteration.

nodejs-github-bot added fs Issues and PRs related to the fs subsystem / file system. needs-ci PRs that need a full CI run. labels Dec 29, 2021

lpinca approved these changes Dec 29, 2021

View reviewed changes

bnb approved these changes Dec 29, 2021

View reviewed changes

mcollina approved these changes Dec 29, 2021

View reviewed changes

lpinca reviewed Dec 29, 2021

View reviewed changes

lib/internal/fs/cp/cp.js Outdated Show resolved Hide resolved

addaleax approved these changes Dec 29, 2021

View reviewed changes

Update lib/internal/fs/cp/cp.js

625c12f

Co-authored-by: Luigi Pinca <luigipinca@gmail.com>

lpinca added the request-ci Add this label to start a Jenkins CI on a PR. label Dec 29, 2021

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Dec 29, 2021

This comment has been minimized.

Sign in to view

Mesteery added the commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. label Dec 29, 2021

cjihrig changed the title ~~fs: use async directory processing in cp()~~ fs: use streaming directory processing in cp() Dec 29, 2021

juanarbol approved these changes Dec 29, 2021

View reviewed changes

github-actions bot mentioned this pull request Dec 30, 2021

CI Reliability 2021-12-30 nodejs/reliability#160

Open

27 tasks

Mesteery added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Dec 30, 2021

VoltrexKeyva approved these changes Dec 30, 2021

View reviewed changes

github-actions bot mentioned this pull request Dec 31, 2021

CI Reliability 2021-12-31 nodejs/reliability#161

Open

26 tasks

cjihrig added the commit-queue Add this label to land a pull request using GitHub Actions. label Dec 31, 2021

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Dec 31, 2021

nodejs-github-bot merged commit d0c1176 into nodejs:master Dec 31, 2021

cjihrig deleted the cp branch December 31, 2021 20:47

This was referenced Jan 1, 2022

CI Reliability 2022-01-01 nodejs/reliability#162

Open

CI Reliability 2022-01-02 nodejs/reliability#163

Open

CI Reliability 2022-01-03 nodejs/reliability#164

Open

This was referenced Jan 4, 2022

CI Reliability 2022-01-04 nodejs/reliability#165

Open

CI Reliability 2022-01-05 nodejs/reliability#166

Open

CI Reliability 2022-01-06 nodejs/reliability#167

Open

CI Reliability 2022-01-07 nodejs/reliability#168

Open

targos mentioned this pull request Jan 16, 2022

v17.4.0 release proposal #41557

Merged

danielleadams mentioned this pull request Feb 1, 2022

v16.14.0 proposal #41804

Merged

RyanZim mentioned this pull request Oct 21, 2022

Use fs.opendir() in copy*() jprichardson/node-fs-extra#972

Closed

SukkaW added a commit to SukkaW/node-fs-extra that referenced this pull request Nov 28, 2023

refactor(copy): backport nodejs/node#41351

ac14e53

SukkaW added a commit to SukkaW/node-fs-extra that referenced this pull request Nov 28, 2023

refactor(copy): backport nodejs/node#41351

1a80ec5

RyanZim pushed a commit to jprichardson/node-fs-extra that referenced this pull request Feb 10, 2024

Refactor copy to use opendir (#1028)

1d931c8

* refactor(copy): backport nodejs/node#41351 * perf(copy): parallel copy * perf(copy): run filter in parallel as well

vrecomet added a commit to vrecomet/Metaln00b1 that referenced this pull request May 17, 2025

Refactor copy to use opendir (#1028)

5d11ef0

* refactor(copy): backport nodejs/node#41351 * perf(copy): parallel copy * perf(copy): run filter in parallel as well

baiiiiiiix pushed a commit to baiiiiiiix/Documentation that referenced this pull request May 17, 2025

Refactor copy to use opendir (#1028)

9065de5

* refactor(copy): backport nodejs/node#41351 * perf(copy): parallel copy * perf(copy): run filter in parallel as well

neysidevhcr1j added a commit to neysidevhcr1j/neysidevhcr1j that referenced this pull request May 20, 2025

Refactor copy to use opendir (#1028)

b7c5b9c

* refactor(copy): backport nodejs/node#41351 * perf(copy): parallel copy * perf(copy): run filter in parallel as well

KennetYoung added a commit to KennetYoung/ingeni that referenced this pull request Jun 10, 2025

Refactor copy to use opendir (#1028)

085671e

* refactor(copy): backport nodejs/node#41351 * perf(copy): parallel copy * perf(copy): run filter in parallel as well

criticalPathway pushed a commit to criticalPathway/node-fs-extra that referenced this pull request Nov 17, 2025

Refactor copy to use opendir (#1028)

df48bad

* refactor(copy): backport nodejs/node#41351 * perf(copy): parallel copy * perf(copy): run filter in parallel as well

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fs: use streaming directory processing in cp()#41351

fs: use streaming directory processing in cp()#41351
nodejs-github-bot merged 2 commits intonodejs:masterfrom
cjihrig:cp

cjihrig commented Dec 29, 2021

Uh oh!

bnb commented Dec 29, 2021

Uh oh!

mcollina left a comment

Uh oh!

Uh oh!

cjihrig commented Dec 29, 2021

Uh oh!

This comment has been minimized.

nodejs-github-bot commented Dec 29, 2021

Uh oh!

juanarbol left a comment

Uh oh!

nodejs-github-bot commented Dec 31, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Uh oh!

Conversation

cjihrig commented Dec 29, 2021

Uh oh!

bnb commented Dec 29, 2021

Uh oh!

mcollina left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cjihrig commented Dec 29, 2021

Uh oh!

This comment has been minimized.

nodejs-github-bot commented Dec 29, 2021

Uh oh!

juanarbol left a comment

Choose a reason for hiding this comment

Uh oh!

nodejs-github-bot commented Dec 31, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants