[POC] Show 'progress' of test in terminal, animating every 100ms

[Sajjon]

Show 'progress' of test in terminal, animating every 100ms .

I dont expect this PR to be merged, thus creating it as DRAFT, my intention is to start a discussion about if this is something we want.

Motivation:

swift-test is already much much more pretty and FUN than XCTest, I love the color and symbols! In fact, in every Swift team I've been working in since the release of Swift, the majority of colleagues have loved emojis.

Yarn is just... fun and beautiful to use! The value is NOT to underestimate, making testing fun, is beneficial for all mankind.

it is not so useful to read

Test Foo started.
Test Foo passed.
Test Bar started.
Test Bar passed.

Better is:

Test Foo passed.
Test Bar passed.

if they both passed, but still we wanna see when a test starts... so solution is to use what yarn does with clearline, to replace the message "Test Foo started." with "Test Foo passed.".

This PR not only does, that, but it adds animated progress!

Modifications:

Adding another Event case testProgressTick(Int) which is emitted by the test runner every 150ms (not completely arbitrarily set..., it resulted in smooth animation), this is just a placeholder implementation... a real implementation should maybe piggy back on the timeout logics Clock etc.

Result:

From tiny POC (using this PRs source branch)

Future direction

A real implementation would only use this if:

• useANSIEscapeCodes is specified
• isParallelizationEnabled is false
• A new Event.Recorder.Option caseuseAnimation (or similar name) would be true

The events testCaseStarted and testCaseEnded would also need to be handled.

[grynspan]

This is an interesting PR, but I think we might run into some difficulties pretty quickly:

• We tried to design Event.Recorder as terminal-agnostically as possible. We would want any new uses of ANSI control codes to be tested and confirmed-functional on macOS' Terminal.app, Ubuntu's Terminal, pure-CLI Linux, and Windows' modern Console.
• We need to make sure that output is legible when written to something like Jenkins or a plain text file, but I think with this change the amount of data written to output is significantly increased.
• It is important to avoid the use of unstructured tasks (Task) because not all target platforms support them. Task groups are usually sufficient.
• We avoided using emoji in our console output because they render very differently in different environments and don't render correctly at all in pure-CLI Linux. Are there Unicode characters you could use instead (box drawing maybe)? Are there SF Symbols that could be used instead on macOS when SF Pro is installed?
• If a test writes to stdout/stderr, it's going to break this rendering, right?

[hassila]

Also one nit: the delay 7s output is removed and replaced with the summary, I'd expect individual output for all threee tests + the summary as a separate row, not replacing the final test result.

[hassila]

For this specific point:

We need to make sure that output is legible when written to something like Jenkins or a plain text file, but I think with this change the amount of data written to output is significantly increased.

We added a --no-progress option to package-benchmark to cater for CI or plain text capture but show a progress bar by default (thus one can suppress it when desired) - similar use case.

[MaxDesiatov]

• We avoided using emoji in our console output because they render very differently in different environments and don't render correctly at all in pure-CLI Linux. Are there Unicode characters you could use instead (box drawing maybe)? Are there SF Symbols that could be used instead on macOS when SF Pro is installed?

Another point here is accessibility regression. I imagine a full moon emoji in this context will be quite confusing with a screen reader.

Additionally, I'm curious how SF Symbols would render on non-Darwin platforms?

[grynspan]

Additionally, I'm curious how SF Symbols would render on non-Darwin platforms?

The package's support for SF Symbols is limited to macOS. Other platforms get Unicode characters. See _Symbol in the repo for examples.

[grynspan]

When tests run in parallel, they will be presumably unable to use this trick because the order of output is non-deterministic, and the position of the cursor is not reliably aligned with output for the current task/test.

[Sajjon]

Here are some animations using SF symbols (this is using another branch called "sfsymbols" (which is a complete mess code wise)), I was so happy with the last once I call radiance, that I included three different colours of it.

[Sajjon]

This is an interesting PR, but I think we might run into some difficulties pretty quickly:
Indeed, this PR in its current form is completely unmergable of course :) it is just a POC getting a conversation started :).

So for now ignore implementation, assuming an implementation is:

1. keeps Event.Recorder as terminal-agnostic as possible
2. does not change current (main branch) output behaviour when used on Jenkins/plain text file
3. Does not use unstructured tasks (Task)
4. Uses SF Symbols, never emoji

With nice clean code etc etc, if all of the above were true - what are your thoughts on introducing this feature? Is it something you see fits nicely into the goals of swift-testing?

If a test writes to stdout/stderr, it's going to break this rendering, right?
This is the biggest challenge I think, but it might be solvable...? But solving this challenge is irrelevant if you won't accept a well-written implementation (fulfilling above mentioned requirements).

So basically, please advice :) I don't wanna spend too much time on the biggest challenge if a well implemented PR will be closed anyway.

Thx for the interest so far!

[stmontgomery]

Thanks for putting this PoC together, @Sajjon!

what are your thoughts on introducing this feature? Is it something you see fits nicely into the goals of swift-testing?

At a conceptual level I absolutely think this is valuable and fits within the scope of this project's goals. I have used interactive CLI testing tools like this elsewhere which show live progress as they run and found them very useful since they give the user a more responsive look at what the test runner is doing at any particular moment. I basically see it as a terminal analogue of what some full-featured IDEs provide while running tests.

That said, I definitely agree with others who point out that in its current implementation, this will interact poorly with "regular" stdout/stderr emitted from the test runner process. We know from past experience maintaining XCTest that stdio output is important for many users to see and be able to attribute to a specific test. (There may be other code-level concerns about the specific formatting, use of emojis/symbols, etc. of this draft PR, but the interaction with stdio is the biggest challenge IMO.)

Personally, I think the right architecture to (eventually) achieve this is to structure things into two processes: One "harness" process and one "runner" process. The user directly invokes the harness, and the harness (in turn) spawns a runner process which is where test code executes. The runner process sends its events back to the harness in a structured format using an IPC mechanism, and the harness is free to redirect the stdout & stderr file handles from the runner process anywhere it wants. If we have this two-process model, then the harness process could be where we implement a "live progress" output similar to what you've explored here, and it could capture the stdio from the runner into a file on disk, or somewhere other than the console, to avoid interrupting the live progress. And all of this functionality could be controlled by user options, so in non-interactive scenarios (like in CI) it could be disabled and output from runners passed directly through.

If we decide the approach I described above is worthwhile, I think we'll need to explore how best to achieve this with the swift test CLI tool in particular. For now, I see this PR as a very inspiring PoC showing some exciting ideas for what the console output of swift-testing could look like, but (as you noted above) I don't expect we will merge it in its current form.

[Sajjon]

@stmontgomery sorry for not getting back to you, thank you for you long response, that sounds great. I'm closing this Draft PR, it was only meant to kick off a discussion. The design with harness and runner sounds great!

Gonna leave some suggestions of SF symbols that resulted in nice animation, before closing it:
(as seen above)

SF Symbols

Radiance

  var radiance: [Character] {
    [
      "\u{1001ef}",  
      "\u{1001f1}",  
      "\u{1001f2}",  
      "\u{101239}",  
      "\u{101485}",  
      "\u{101239}",  // repeat
      "\u{1001f2}",  
      "\u{1001f1}",  
    ]
  }

Moon

var moon: [Character] { [
        "\u{101409}", 
        "\u{10140a}", 
        "\u{10140b}", 
        "\u{10140c}", 
        "\u{10140d}", 
        "\u{10140e}", 
        "\u{10140f}", 
        "\u{101410}", 
      ]
}

Numpad

var numPad: [Character] { [
        "\u{1009b2}", 
        "\u{1009b4}", 
        "\u{1009b5}", 
        "\u{1009b6}",
        "\u{1009b7}", 
        "\u{1009b8}", 
        "\u{1009b9}", 
        "\u{1009ba}", 
        "\u{1009bb}", 
        "\u{1009bc}", 
    ]
}

Unicode

I think one can probably make something nice out of braille unicode series, it is quite a long series, and I think one can find some nice subseries in it, i.e. tick % someNum, to make is so that the between two consecutive unicodes the dot only either moves one position, or we only increase dot count by one, but never decrease it.

Braile

var braille: [Character] { [
        "\u{2800}", // ⠀
        "\u{2801}", // ⠁
        "\u{2802}", // ⠂
        "\u{2803}", // ⠃
        "\u{2804}", // ⠄
        "\u{2805}", // ⠅
        "\u{2806}", // ⠆
        "\u{2807}", // ⠇
        "\u{2808}", // ⠈
        "\u{2809}", // ⠉
        "\u{280a}", // ⠊
        "\u{280b}", // ⠋
        "\u{280c}", // ⠌
        "\u{280d}", // ⠍
        "\u{280e}", // ⠎
        "\u{280f}", // ⠏
        "\u{2810}", // ⠐
        "\u{2811}", // ⠑
        "\u{2812}", // ⠒
        "\u{2813}", // ⠓
        "\u{2814}", // ⠔
        "\u{2815}", // ⠕
        "\u{2816}", // ⠖
        "\u{2817}", // ⠗
        "\u{2818}", // ⠘
        "\u{2819}", // ⠙
        "\u{281a}", // ⠚
        "\u{281b}", // ⠛
        "\u{281c}", // ⠜
        "\u{281d}", // ⠝
        "\u{281e}", // ⠞
        "\u{281f}", // ⠟
        "\u{2820}", // ⠠
        "\u{2821}", // ⠡
        "\u{2822}", // ⠢
        "\u{2823}", // ⠣
        "\u{2824}", // ⠤
        "\u{2825}", // ⠥
        "\u{2826}", // ⠦
        "\u{2827}", // ⠧
        "\u{2828}", // ⠨
        "\u{2829}", // ⠩
        "\u{282a}", // ⠪
        "\u{282b}", // ⠫
        "\u{282c}", // ⠬
        "\u{282d}", // ⠭
        "\u{282e}", // ⠮
        "\u{282f}", // ⠯
        "\u{2830}", // ⠰
        "\u{2831}", // ⠱
        "\u{2832}", // ⠲
        "\u{2833}", // ⠳
        "\u{2834}", // ⠴
        "\u{2835}", // ⠵
        "\u{2836}", // ⠶
        "\u{2837}", // ⠷
        "\u{2838}", // ⠸
        "\u{2839}", // ⠹
        "\u{283a}", // ⠺
        "\u{283b}", // ⠻
        "\u{283c}", // ⠼
        "\u{283d}", // ⠽
        "\u{283e}", // ⠾
        "\u{283f}", // ⠿
      ]
}