Merge pull request #12 from tannewt/v4

Add versions 3 and 4

Author: antonio-openroad

README.rst

@@ -87,9 +87,18 @@ The base UUID used in characteristics is ``ADAFxxxx-4669-6C65-5472-616E73666572`
 
 The service has two characteristics:
 
-* version (``0x0100``) - Simple unsigned 32-bit integer version number. Always 1.
+* version (``0x0100``) - Simple unsigned 32-bit integer version number. May be 1 - 4.
 * raw transfer (``0x0200``) - Bidirectional link with a custom protocol. The client does WRITE_NO_RESPONSE to the characteristic and then server replies via NOTIFY. (This is similar to the Nordic UART Service but on a single characteristic rather than two.) The commands over the transfer characteristic are idempotent and stateless. A disconnect during a command will reset the state.
 
+Time resolution
+---------------
+
+Time resolution varies based filesystem type. FATFS can only get down to the 2 second bound after 1980. Littlefs can do 64-bit nanoseconds after January 1st, 1970.
+
+To account for this, the protocol has time in 64-bit nanoseconds after January 1st, 1970. However, the server will respond with a potentially truncated version that is the value stored.
+
+Also note that devices serving the file transfer protocol may not have it's own clock so do not rely on time ordering. Any internal writes may set the time incorrectly. So, we only recommend using the value as a cache key.
+
 Commands
 ---------
 
@@ -151,6 +160,7 @@ The header is four fixed entries and a variable length path:
 * 1 Byte reserved for padding.
 * Path length: 16-bit number encoding the encoded length of the path string.
 * Offset: 32-bit number encoding the starting offset to write.
+* Current time: 64-bit number encoding nanoseconds since January 1st, 1970. Used as the file modification time. Not all system will support the full resolution. Use the truncated time response value for caching.
 * Total size: 32-bit number encoding the total length of the file contents.
 * Path: UTF-8 encoded string that is *not* null terminated. (We send the length instead.)
 
@@ -160,6 +170,7 @@ The server will repeatedly respond until the total length has been transferred w
 * Status: Single byte. ``0x01`` if OK. ``0x02`` if any parent directory is missing or a file.
 * 2 Bytes reserved for padding.
 * Offset: 32-bit number encoding the starting offset to write. (Should match the offset from the previous 0x20 or 0x22 message)
+* Truncated time: 64-bit number encoding nanoseconds since January 1st, 1970 as stored by the file system. The resolution may be less that the protocol. It is sent back for use in caching on the host side.
 * Free space: 32-bit number encoding the amount of data the client can send.
 
 The client will repeatedly respond until the total length has been transferred with:
@@ -173,11 +184,13 @@ The client will repeatedly respond until the total length has been transferred w
 
 The transaction is complete after the server has received all data and replied with a status with 0 free space and offset set to the content length.
 
+**NOTE**: Current time was added in version 3. The rest of the packets remained the same.
+
 
 ``0x30`` - Delete a file or directory
 +++++++++++++++++++++++++++++++++++++
 
-Deletes the file or directory at the given full path. Directories must be empty to be deleted.
+Deletes the file or directory at the given full path. Non-empty directories will have their contents deleted as well.
 
 The header is two fixed entries and a variable length path:
 
@@ -189,7 +202,9 @@ The header is two fixed entries and a variable length path:
 The server will reply with:
 
 * Command: Single byte. Always ``0x31``.
-* Status: Single byte. ``0x01`` if the file or directory was deleted or ``0x02`` if the path is a non-empty directory or non-existent.
+* Status: Single byte. ``0x01`` if the file or directory was deleted or ``0x02`` if the path is non-existent.
+
+**NOTE**: In version 2, this command now deletes contents of a directory as well. It won't error.
 
 ``0x40`` - Make a directory
 +++++++++++++++++++++++++++
@@ -201,12 +216,16 @@ The header is two fixed entries and a variable length path:
 * Command: Single byte. Always ``0x40``.
 * 1 Byte reserved for padding.
 * Path length: 16-bit number encoding the encoded length of the path string.
+* 4 Bytes reserved for padding.
+* Current time: 64-bit number encoding nanoseconds since January 1st, 1970. Used as the file modification time. Not all system will support the full resolution. Use the truncated time response value for caching.
 * Path: UTF-8 encoded string that is *not* null terminated. (We send the length instead.)
 
 The server will reply with:
 
 * Command: Single byte. Always ``0x41``.
 * Status: Single byte. ``0x01`` if the directory(s) were created or ``0x02`` if any parent of the path is an existing file.
+* 6 Bytes reserved for padding.
+* Truncated time: 64-bit number encoding nanoseconds since January 1st, 1970 as stored by the file system. The resolution may be less that the protocol. It is sent back for use in caching on the host side.
 
 ``0x50`` - List a directory
 +++++++++++++++++++++++++++
@@ -232,11 +251,53 @@ The server will reply with n+1 entries for a directory with n files:
   - Bit 0: Set when the entry is a directory
   - Bits 1-7: Reserved
 
+* Modification time: 64-bit number of nanoseconds since January 1st, 1970. *However*, files modifiers may not have an accurate clock so do *not* assume it is correct. Instead, only use it to determine cacheability vs a local copy.
 * File size: 32-bit number encoding the size of the file. Ignore for directories. Value may change.
 * Path: UTF-8 encoded string that is *not* null terminated. (We send the length instead.) These paths are relative so they won't contain ``/`` at all.
 
 The transaction is complete when the final entry is sent from the server. It will have entry number == total entries and zeros for flags, file size and path length.
 
+``0x60`` - Move a file or directory
++++++++++++++++++++++++++++++++++++
+
+Moves a file or directory at a given path to a different path. Can be used to
+rename as well. The two paths are sent separated by a byte so that the server
+may null-terminate the string itself. The client may send anything there.
+
+The header is two fixed entries and a variable length path:
+
+* Command: Single byte. Always ``0x60``.
+* 1 Byte reserved for padding.
+* Old Path length: 16-bit number encoding the encoded length of the path string.
+* New Path length: 16-bit number encoding the encoded length of the path string.
+* Old Path: UTF-8 encoded string that is *not* null terminated. (We send the length instead.)
+* One padding byte. This can be used to null terminate the old path string.
+* New Path: UTF-8 encoded string that is *not* null terminated. (We send the length instead.)
+
+The server will reply with:
+* Command: Single byte. Always ``0x61``.
+* Status: Single byte. ``0x01`` on success or ``0x02`` on error.
+
+**NOTE**: This is added in version 4.
+
+Versions
+=========
+
+Version 2
+---------
+* Changes delete to delete contents of non-empty directories automatically.
+
+Version 3
+---------
+* Adds modification time.
+  * Adds current time to file write command.
+  * Adds current time to make directory command.
+  * Adds modification time to directory listing entries.
+
+Version 4
+---------
+* Adds move command.
+
 Contributing
 ============
 

adafruit_ble_file_transfer.py

@@ -13,6 +13,7 @@
 """
 
 import struct
+import time
 import _bleio
 
 from adafruit_ble.attributes import Attribute
@@ -75,7 +76,7 @@ class FileTransferService(Service):
     # pylint: disable=too-few-public-methods
 
     uuid = StandardUUID(0xFEBB)
-    version = Uint32Characteristic(uuid=FileTransferUUID(0x0100))
+    version = Uint32Characteristic(uuid=FileTransferUUID(0x0100), initial_value=4)
     raw = _TransferCharacteristic()
     # _raw gets shadowed for each MIDIService instance by a PacketBuffer. PyLint doesn't know this
     # so it complains about missing members.
@@ -95,6 +96,8 @@ class FileTransferService(Service):
     MKDIR_STATUS = 0x41
     LISTDIR = 0x50
     LISTDIR_ENTRY = 0x51
+    MOVE = 0x60
+    MOVE_STATUS = 0x61
 
     # Responses
     # 0x00 is INVALID
@@ -117,6 +120,9 @@ class FileTransferClient:
     def __init__(self, service):
         self._service = service
 
+        if service.version < 3:
+            raise RuntimeError("Service on other device too old")
+
     def _write(self, buffer):
         # print("write", binascii.hexlify(buffer))
         sent = 0
@@ -135,7 +141,7 @@ def _readinto(self, buffer):
                 read = self._service.raw.readinto(buffer)
             except ValueError:
                 read = self._service.raw.readinto(long_buffer)
-        buffer[:read] = long_buffer[:read]
+                buffer[:read] = long_buffer[:read]
         return read
 
     def read(self, path, *, offset=0):
@@ -204,24 +210,32 @@ def read(self, path, *, offset=0):
             self._write(encoded)
         return buf
 
-    def write(self, path, contents, *, offset=0):
+    def write(self, path, contents, *, offset=0, modification_time=None):
         """Writes the given contents to the given path starting at the given offset.
+        Returns the trunctated modification time.
 
         If the file is shorter than the offset, zeros will be added in the gap."""
         path = path.encode("utf-8")
         total_length = len(contents) + offset
+        if modification_time is None:
+            modification_time = int(time.time() * 1_000_000_000)
         encoded = (
             struct.pack(
-                "<BxHII", FileTransferService.WRITE, len(path), offset, total_length
+                "<BxHIQI",
+                FileTransferService.WRITE,
+                len(path),
+                offset,
+                modification_time,
+                total_length,
             )
             + path
         )
         self._write(encoded)
-        b = bytearray(struct.calcsize("<BBxxII"))
+        b = bytearray(struct.calcsize("<BBxxIQI"))
         written = 0
         while written < len(contents):
             self._readinto(b)
-            cmd, status, current_offset, free_space = struct.unpack("<BBxxII", b)
+            cmd, status, current_offset, _, free_space = struct.unpack("<BBxxIQI", b)
             if status != FileTransferService.OK:
                 print("write error", status)
                 raise RuntimeError()
@@ -254,23 +268,32 @@ def write(self, path, contents, *, offset=0):
 
         # Wait for confirmation that everything was written ok.
         self._readinto(b)
-        cmd, status, offset, free_space = struct.unpack("<BBxxII", b)
+        cmd, status, offset, truncated_time, free_space = struct.unpack("<BBxxIQI", b)
         if cmd != FileTransferService.WRITE_PACING or offset != total_length:
             raise ProtocolError()
+        return truncated_time
 
-    def mkdir(self, path):
-        """Makes the directory and any missing parents"""
+    def mkdir(self, path, modification_time=None):
+        """Makes the directory and any missing parents. Returns the truncated time"""
         path = path.encode("utf-8")
-        encoded = struct.pack("<BxH", FileTransferService.MKDIR, len(path)) + path
+        if modification_time is None:
+            modification_time = int(time.time() * 1_000_000_000)
+        encoded = (
+            struct.pack(
+                "<BxHxxxxQ", FileTransferService.MKDIR, len(path), modification_time
+            )
+            + path
+        )
         self._write(encoded)
 
-        b = bytearray(struct.calcsize("<BB"))
+        b = bytearray(struct.calcsize("<BBxxxxxxQ"))
         self._readinto(b)
-        cmd, status = struct.unpack("<BB", b)
+        cmd, status, truncated_time = struct.unpack("<BBxxxxxxQ", b)
         if cmd != FileTransferService.MKDIR_STATUS:
             raise ProtocolError()
         if status != FileTransferService.OK:
             raise ValueError("Invalid path")
+        return truncated_time
 
     def listdir(self, path):
         """Returns a list of tuples, one tuple for each file or directory in the given path"""
@@ -282,31 +305,33 @@ def listdir(self, path):
         b = bytearray(self._service.raw.incoming_packet_length)
         i = 0
         total = 10  # starting value that will be replaced by the first response
-        header_size = struct.calcsize("<BBHIIII")
+        header_size = struct.calcsize("<BBHIIIQI")
         path_length = 0
         encoded_path = b""
+        file_size = 0
+        flags = 0
+        modification_time = 0
         while i < total:
             read = self._readinto(b)
             offset = 0
-            file_size = 0
-            flags = 0
             while offset < read:
                 if len(encoded_path) == path_length:
                     if path_length > 0:
                         path = str(
                             encoded_path,
                             "utf-8",
                         )
-                        paths.append((path, file_size, flags))
+                        paths.append((path, file_size, flags, modification_time))
                     (
                         cmd,
                         status,
                         path_length,
                         i,
                         total,
                         flags,
+                        modification_time,
                         file_size,
-                    ) = struct.unpack_from("<BBHIIII", b, offset=offset)
+                    ) = struct.unpack_from("<BBHIIIQI", b, offset=offset)
                     offset += header_size
                     encoded_path = b""
                     if cmd != FileTransferService.LISTDIR_ENTRY:
@@ -322,7 +347,7 @@ def listdir(self, path):
         return paths
 
     def delete(self, path):
-        """Deletes the file or directory at the given path. Directories must be empty."""
+        """Deletes the file or directory at the given path."""
         path = path.encode("utf-8")
         encoded = struct.pack("<BxH", FileTransferService.DELETE, len(path)) + path
         self._write(encoded)
@@ -334,3 +359,25 @@ def delete(self, path):
             raise ProtocolError()
         if status != FileTransferService.OK:
             raise ValueError("Missing file")
+
+    def move(self, old_path, new_path):
+        """Moves the file or directory from old_path to new_path."""
+        if self._service.version < 4:
+            raise RuntimeError("Service on other device too old")
+        old_path = old_path.encode("utf-8")
+        new_path = new_path.encode("utf-8")
+        encoded = (
+            struct.pack("<BxHH", FileTransferService.MOVE, len(old_path), len(new_path))
+            + old_path
+            + b" "
+            + new_path
+        )
+        self._write(encoded)
+
+        b = bytearray(struct.calcsize("<BB"))
+        self._readinto(b)
+        cmd, status = struct.unpack("<BB", b)
+        if cmd != FileTransferService.MOVE_STATUS:
+            raise ProtocolError()
+        if status != FileTransferService.OK:
+            raise ValueError("Missing file")

examples/ble_file_transfer_listdirs.py

@@ -0,0 +1,54 @@
+# SPDX-FileCopyrightText: 2020 ladyada for Adafruit Industries
+# SPDX-License-Identifier: MIT
+
+"""
+Used with ble_uart_echo_test.py. Transmits "echo" to the UARTService and receives it back.
+"""
+
+import sys
+
+from adafruit_ble import BLERadio
+from adafruit_ble.advertising.standard import (
+    ProvideServicesAdvertisement,
+    Advertisement,
+)
+import adafruit_ble_file_transfer
+
+# Connect to a file transfer device
+ble = BLERadio()
+connection = None
+print("disconnected, scanning")
+for advertisement in ble.start_scan(
+    ProvideServicesAdvertisement, Advertisement, timeout=1
+):
+    # print(advertisement.address, advertisement.address.type)
+    if (
+        not hasattr(advertisement, "services")
+        or adafruit_ble_file_transfer.FileTransferService not in advertisement.services
+    ):
+        continue
+    connection = ble.connect(advertisement)
+    peer_address = advertisement.address
+    print("connected to", advertisement.address)
+    break
+ble.stop_scan()
+
+if not connection:
+    print("No advertisement found")
+    sys.exit(1)
+
+# Prep the connection
+if adafruit_ble_file_transfer.FileTransferService not in connection:
+    print("Connected device missing file transfer service")
+    sys.exit(1)
+if not connection.paired:
+    print("pairing")
+    connection.pair()
+print("paired")
+print()
+service = connection[adafruit_ble_file_transfer.FileTransferService]
+client = adafruit_ble_file_transfer.FileTransferClient(service)
+
+# Do the file operations
+print(client.listdir("/"))
+print(client.listdir("/lib/"))