The Monterey Merge #3058

millenomi · 2021-08-23T20:22:52Z

This PR brings the FOSS portions of Core Foundation on par with with macOS Monterey, iOS 15, watchOS 8 and tvOS 15. This includes the changes shipped during the previous releases.

Most of the changes include bug fixes and performance improvements, but some are useful to highlight:

This patch adds the Core Foundation code that underlies Foundation's ListFormatter and RelativeDateTimeFormatter to the FOSS subset.
Several Emoji fixes are included that update CFString (and thus NSString and the Foundation methods on Swift.String) support for correctly decoding and iterating over emoji introduced during the Big Sur and Monterey timeframes. The corresponding character maps in CharacterSet have been updated.
Certain parsing functions, most notably those that implement property list parsing, now have a maximum recursion depth to prevent resource abuse from malicious input.
Some architectural changes have been introduced that are used on Darwin to support Pointer Authentication. For more information on the topic, see Preparing Your App to Work with Pointer Authentication. Note that the FOSS subset does not ship on architectures that require this support, but the architectural changes have been merged for simplicity and synchronization between Darwin and FOSS builds.
Parameter checking has been improved for several CF functions.
Some implementations have been streamlined or consolidated, especially for CFPlugIn and CFBundle. CFBundle also adds support for "Wrapper"-style bundles (for example, the kind of bundle produced when installing iOS applications on Macs with Apple silicon).

This PR brings the FOSS portions of Core Foundation on par with macOS Monterey, iOS 15, watchOS 8 and tvOS 15. This includes the changes shipped during the previous releases. Most of the changes include bug fixes and performance improvements, but some are useful to highlight: - This patch adds the Core Foundation code that underlies Foundation's ListFormatter and RelativeDateTimeFormatter to the FOSS subset. - Several Emoji fixes are included that update CFString (and thus NSString and the Foundation methods on Swift.String) support for correctly decoding and iterating over emoji introduced during the Big Sur and Monterey timeframes. The corresponding character maps in CharacterSet have been updated. - Certain parsing functions, most notably those that implement property list parsing, now have a maximum recursion depth to prevent resource abuse from malicious input. - Some architectural changes have been introduced that are used on Darwin to support Pointer Authentication. For more information on the topic, see [Preparing Your App to Work with Pointer Authentication](https://developer.apple.com/documentation/security/preparing_your_app_to_work_with_pointer_authentication). Note that the FOSS subset does not ship on architectures that require this support, but the architectural changes have been merged for simplicity and synchronization between Darwin and FOSS builds. - Parameter checking has been improved for several CF functions. - Some implementations have been streamlined or consolidated, especially for CFPlugIn and CFBundle. CFBundle also adds support for "Wrapper"-style bundles (for example, the kind of bundle produced when installing iOS applications on Macs with Apple silicon).

millenomi · 2021-08-23T20:23:48Z

@swift-ci please test

millenomi · 2021-08-23T20:23:55Z

cc @parkera

millenomi · 2021-08-23T20:24:26Z

cc @compnerd — This has a few bits that I know we need Windows expertise on; can we kick testing and see if there's anything that breaks?

compnerd · 2021-08-23T20:27:40Z

Sure, I can kick off a build for this: https://dev.azure.com/compnerd/swift-build/_build/results?buildId=53638&view=results

compnerd · 2021-08-23T20:40:16Z

For _CFGetPathFromFileDescriptor, something along the lines of the following should do the trick:

HANDLE hFile = _get_osfhandle(fd);
WCHAR wszPath[MAX_PATH + 1];
DWORD dwLength = GetFinalPathNameByHandleW(hFile, &wszPath, ARRAY_SIZE(wszPath), FILE_NAME_NORMALIZED);
if (dwLength == 0)
  return false;
/* do `wcstombcs` dance */

millenomi · 2021-08-23T21:03:54Z

I'll let the first run go through so we can do a single pass with other Windows fixes as needed, but definitely will have a follow-up with that.

CoreFoundation/Base.subproj/CFRuntime.c

CoreFoundation/NumberDate.subproj/CFDate.c

- The `&` in the macro definition is redundant for os_unfair_lock_lock_with_options - Remove calls that aren't defined for Swift for Windows. - Restore CFDate initialization code that should be preserved.

millenomi · 2021-08-24T22:46:04Z

@compnerd This is ready for another try on Windows.

millenomi · 2021-08-24T22:46:08Z

@swift-ci please test

millenomi · 2021-08-25T07:22:25Z

cc @drexin This adds dependencies on new ICU symbols, do you see anything that may be alarming?

3405691582 · 2021-08-23T23:55:57Z

CoreFoundation/Collections.subproj/CFBasicHash.c

@@ -1452,7 +1473,6 @@ CF_PRIVATE size_t CFBasicHashGetSize(CFConstBasicHashRef ht, Boolean total) {
    if (ht->bits.keys_offset) size += sizeof(CFBasicHashValue *);
    if (ht->bits.counts_offset) size += sizeof(void *);
    if (__CFBasicHashHasHashCache(ht)) size += sizeof(uintptr_t *);
-#if ENABLE_MEMORY_COUNTERS || ENABLE_DTRACE_PROBES


#2626 needs to be reapplied to condition usage of malloc_size.

3405691582 · 2021-08-23T23:57:22Z

CoreFoundation/Collections.subproj/CFStorage.c

@@ -1371,6 +1381,21 @@ static void __CFStorageApplyNodeBlock(CFStorageRef storage, void (^block)(CFStor
    __CFStorageApplyNodeBlockInterior(storage, &storage->rootNode, block);
 }

+static CFIndex __CFStorageEstimateTotalAllocatedSize(CFStorageRef storage) __attribute__((unused));


#2626 needs to be reapplied.

If we must to keep this, we should use an #ifdef conditional for the use of malloc_size.

3405691582 · 2021-08-23T23:58:59Z

CoreFoundation/PlugIn.subproj/CFBundle_Internal.h

+#if TARGET_OS_MAC || TARGET_OS_BSD
+
+static bool _CFGetPathFromFileDescriptor(int fd, char *path) {
+    return fcntl(fd, F_GETPATH, path) != -1;


OpenBSD doesn't have F_GETPATH (FreeBSD and NetBSD do however). #ifdef F_GETPATH ... #else #warning ... return false; #endif would suffice as buildfix for now.

We already have a generic fallback; I'll just remove the TARGET_OS_BSD.

… or condition this to ! __OpenBSD__.

3405691582 · 2021-08-24T00:07:48Z

CoreFoundation/PlugIn.subproj/CFBundle_Resources.c

@@ -26,6 +26,10 @@
 #include <errno.h>
 #include <sys/types.h>

+#if !TARGET_OS_MAC && !TARGET_OS_BSD


FreeBSD and NetBSD have this, OpenBSD does not. This needs to be #if (!TARGET_OS_MAC && !TARGET_OS_BSD) || defined(__OpenBSD__) or similar.

3405691582 · 2021-08-24T00:10:53Z

CoreFoundation/URL.subproj/CFURL.c

@@ -26,6 +25,7 @@
 #if TARGET_OS_OSX
 #include <CoreFoundation/CFNumberFormatter.h>
 #endif
+#include <assert.h>


This necessitates adding || TARGET_OS_BSD in the conditional on l24.

3405691582 · 2021-08-26T13:34:07Z

CoreFoundation/CMakeLists.txt

@@ -60,6 +60,7 @@ if(NOT "${CMAKE_C_SIMULATE_ID}" STREQUAL "MSVC")
  add_compile_options($<$<COMPILE_LANGUAGE:C>:-fconstant-cfstrings>)
  add_compile_options($<$<COMPILE_LANGUAGE:C>:-fdollars-in-identifiers>)
  add_compile_options($<$<COMPILE_LANGUAGE:C>:-fno-common>)
+  add_compile_options($<$<COMPILE_LANGUAGE:C>:-Werror=implicit-function-declaration>)


Now that we're doing this, something like

#if __has_include(<pthread_np.h>) #include <pthread_np.h> #endif

or similar is necessary in CFInternal.h for OpenBSD.

YOCKOW · 2021-09-05T01:28:39Z

@millenomi
Sorry for a little digressing.
You said, just one year ago, that small CF changes were temporarily stopped from being merged. However, they are not merged yet as of now.
When should (or shouldn't) such changes be merged?

millenomi · 2021-09-06T19:17:55Z

It's been a hard year. The hope was to do this merge earlier, but we ended up skipping a release.

Once this patch is merged, by and large.

millenomi · 2021-09-10T17:49:54Z

@3405691582 I applied your suggestions, can you give another quick look?

millenomi · 2021-09-10T17:50:07Z

@swift-ci please test

millenomi · 2021-09-10T21:01:28Z

@3405691582 I think things are okay, and I'm happy to make follow-ups for BSD if needed, but I think my next priority is to close this up for Linux & Windows for now.

3405691582 · 2021-09-10T21:26:43Z

@millenomi I cherrypicked back your changes and it built OK on my end so all is well anyway :) Thanks for applying the fixes!

There are some unrelated buildfixes I'll chase in other PRs.

millenomi · 2021-09-11T00:29:20Z

@3405691582 please ping me on those PRs and I'll expedite them.

3405691582 · 2021-09-11T01:37:14Z

Ah, it looks like my concern was dealt with by #3060 already, so I think we're good.

finagolfin · 2021-09-16T17:35:36Z

CoreFoundation/Base.subproj/CFAvailability.h

 #if (__cplusplus)
 #define CF_OPTIONS(_type, _name) _type _name; enum __CF_OPTIONS_ATTRIBUTES : _type
 #else
-#define CF_OPTIONS(_type, _name) int __CF_OPTIONS_ ## _name; enum __CF_OPTIONS_ATTRIBUTES _name : _type; typedef enum _name _name; enum _name : _type
+#define CF_OPTIONS(_type, _name) enum __CF_OPTIONS_ATTRIBUTES _name : _type _name; enum _name : _type


I had to revert these macro patches when building for Android, finagolfin/swift-android-sdk@dba89d2, or building CoreFoundation fails with this error:

error: non-defining declaration of enumeration with a fixed underlying type is only permitted as a standalone declaration; missing list of enumerators? [-Welaborated-enum-base]

I checked the preprocessed source produced with and without these macro changes and got the following diff:

--- a/cf.i 2021-09-16 07:54:13.941791291 +0000 +++ b/cf.i 2021-09-16 08:08:32.447061538 +0000 @@ -199,7 +199,7 @@ typedef __attribute__((objc_bridge(id))) CFTypeRef CFPropertyListRef; -typedef int __CF_ENUM_CFComparisonResult; enum __attribute__((enum_extensibility(open))) CFComparisonResult : CFIndex; typedef enum CFComparisonResult CFComparisonResult; enum CFComparisonResult : CFIndex { +typedef enum __attribute__((enum_extensibility(open))) CFComparisonResult : CFIndex CFComparisonResult; enum CFComparisonResult : CFIndex { kCFCompareLessThan = -1L, kCFCompareEqualTo = 0, kCFCompareGreaterThan = 1

I don't know what to make of that C gibberish so if anybody else knows how to fix it, I'd appreciate it.

lxbndr reviewed Aug 24, 2021

View reviewed changes

CoreFoundation/Base.subproj/CFRuntime.c Outdated Show resolved Hide resolved

lxbndr reviewed Aug 24, 2021

View reviewed changes

CoreFoundation/NumberDate.subproj/CFDate.c Outdated Show resolved Hide resolved

millenomi added 3 commits August 24, 2021 15:45

Windows build fixes

1aa1cee

- The `&` in the macro definition is redundant for os_unfair_lock_lock_with_options - Remove calls that aren't defined for Swift for Windows. - Restore CFDate initialization code that should be preserved.

Remove a call to a non-defined function.

5c9af60

Restore _CFBundleCopyExecutablePath

71f132a

3405691582 reviewed Aug 26, 2021

View reviewed changes

millenomi added 2 commits September 10, 2021 10:42

Reapply #2626.

00d8416

Take BSD build fixes.

6634ee3

millenomi merged commit 512e412 into swiftlang:main Sep 10, 2021

finagolfin reviewed Sep 16, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

The Monterey Merge #3058

The Monterey Merge #3058

millenomi commented Aug 23, 2021

millenomi commented Aug 23, 2021

millenomi commented Aug 23, 2021

millenomi commented Aug 23, 2021

compnerd commented Aug 23, 2021 •

edited

Loading

compnerd commented Aug 23, 2021

millenomi commented Aug 23, 2021

millenomi commented Aug 24, 2021

millenomi commented Aug 24, 2021

millenomi commented Aug 25, 2021

3405691582 Aug 23, 2021

3405691582 Aug 23, 2021

3405691582 Aug 23, 2021

millenomi Sep 10, 2021

millenomi Sep 10, 2021

3405691582 Aug 24, 2021

3405691582 Aug 24, 2021

3405691582 Aug 26, 2021

YOCKOW commented Sep 5, 2021

millenomi commented Sep 6, 2021

millenomi commented Sep 10, 2021

millenomi commented Sep 10, 2021

millenomi commented Sep 10, 2021

3405691582 commented Sep 10, 2021 •

edited

Loading

millenomi commented Sep 11, 2021

3405691582 commented Sep 11, 2021

finagolfin Sep 16, 2021

The Monterey Merge #3058

The Monterey Merge #3058

Conversation

millenomi commented Aug 23, 2021

millenomi commented Aug 23, 2021

millenomi commented Aug 23, 2021

millenomi commented Aug 23, 2021

compnerd commented Aug 23, 2021 • edited Loading

compnerd commented Aug 23, 2021

millenomi commented Aug 23, 2021

millenomi commented Aug 24, 2021

millenomi commented Aug 24, 2021

millenomi commented Aug 25, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YOCKOW commented Sep 5, 2021

millenomi commented Sep 6, 2021

millenomi commented Sep 10, 2021

millenomi commented Sep 10, 2021

millenomi commented Sep 10, 2021

3405691582 commented Sep 10, 2021 • edited Loading

millenomi commented Sep 11, 2021

3405691582 commented Sep 11, 2021

Choose a reason for hiding this comment

compnerd commented Aug 23, 2021 •

edited

Loading

3405691582 commented Sep 10, 2021 •

edited

Loading