|
| 1 | +--- |
| 2 | +title: "Build Notes: Creating a Local-First, AI-powered YouTube Summarizer" |
| 3 | +date: "2023-12-13" |
| 4 | +--- |
| 5 | + |
| 6 | +My wife and I have been working on a side project for a while — an AI-powered tool that summarizes YouTube videos. |
| 7 | + |
| 8 | +This tool was highly inspired by [https://www.summarize.tech/](https://www.summarize.tech/) — which I used a lot before building my own — so thanks Pete Hunt! |
| 9 | + |
| 10 | +I'm much more of a reader than video watcher so tools like this are perfect when I run across an interesting looking YouTube video. I might still end up watching but the video summary at least quickly satisfies my curiousity. |
| 11 | + |
| 12 | + |
| 13 | + |
| 14 | +This was also an excuse to build something with LLMs and Local-First tools ([ElectricSQL](https://electric-sql.com/) in this case). If you’re new to [Local-First — check out my explainer post from a few months ago](https://bricolage.io/some-notes-on-local-first-development/). |
| 15 | + |
| 16 | +Check it out at [https://samurize.shannon-soper.com](https://samurize.shannon-soper.com) and then read on for some build notes. |
| 17 | + |
| 18 | +### Table of Contents |
| 19 | + |
| 20 | +```toc |
| 21 | +# This code block gets replaced with the TOC |
| 22 | +exclude: Table of Contents |
| 23 | +``` |
| 24 | + |
| 25 | + |
| 26 | +## Lazy syncing and pre-running route queries for fast & smooth route transitions |
| 27 | + |
| 28 | +ElectricSQL syncs data between a backend Postgres database and client SQLite databases. Instead of loading route data through a backend API, you just run reactive SQL queries against the local database! |
| 29 | + |
| 30 | +Which is awesome! I love having the full power of SQL to query data + built-in reactivity for real-time updates. It’s really everything I’ve ever wanted for client-side data. |
| 31 | + |
| 32 | +But there’s two issues before your reactive queries can go to work. First is ensuring the data you want to query is synced. And the second is pre-running the query so the new route can immediately render. |
| 33 | + |
| 34 | + |
| 35 | +### Route Syncing |
| 36 | + |
| 37 | +The simplest way to build a local-first app is to just sync upfront all data but this gets slow as the data size grows. So just like with code splitting, you can split data syncing along route boundaries so the user waits for only the minimal amount of data to be synced. |
| 38 | + |
| 39 | +It’d be really rare to want an entire database synced to each client. So instead you can specify which tables and even which subsets of tables — e.g. only load the latest 10 notifications. |
| 40 | + |
| 41 | +ElectricSQL has this concept of “[Shapes](https://electric-sql.com/docs/usage/data-access/shapes)” — which let you declare the shape of data you want synced to construct a particular route's UI. It’s basically the declarative equivalent of making an API call (an imperative operation). Instead of saying “fetch this shape of data”, you say “sync this shape of data”. You get the same initial load but ElectricSQL also ensures any updates across the system continue to get synced to you in real-time. |
| 42 | + |
| 43 | +This btw is basically jQuery -> React again — jQuery made you push the DOM around and add event listeners, etc. — work we don’t have to do with React and other new reactive JS frameworks. And in the same sense, sync engines like ElectricSQL give you a real-time reactive data system for your entire stack. Your client can declare the data it needs and ElectricSQL makes it so. |
| 44 | + |
| 45 | +So shapes are great but the next question is where to put them. The route loader function (in React Router) seems to me the obvious place and what I did with Samurize. |
| 46 | + |
| 47 | +I [proposed a `SyncManager` API for this to ElectricSQL](https://github.com/electric-sql/electric/discussions/704) that looks like: |
| 48 | + |
| 49 | + |
| 50 | +```ts |
| 51 | +{ |
| 52 | + path: `/video/:videoId`, |
| 53 | + element: <Video />, |
| 54 | + loader: async ({ params }) => { |
| 55 | + await syncManager.load([ |
| 56 | + { |
| 57 | + shape: () => |
| 58 | + db.users.sync({ |
| 59 | + where: { id: params.user_id }, |
| 60 | + }), |
| 61 | + isDone: (shapeMetatdata) => |
| 62 | + // Allow data that's up to 12 hours stale. |
| 63 | + shapeMetadata.msSinceLastSynced < 12 * 60 * 60 * 1000 || |
| 64 | + shapeMetadata.state === `SYNCED`, |
| 65 | + }, |
| 66 | + { |
| 67 | + shape: () => |
| 68 | + db.youtube_videos.sync({ |
| 69 | + where: { id: params.video_id }, |
| 70 | + }), |
| 71 | + isDone: (shapeMetatdata) => |
| 72 | + // Allow data that's up to 12 hours stale. |
| 73 | + shapeMetadata.msSinceLastSynced < 12 * 60 * 60 * 1000 || |
| 74 | + shapeMetadata.state === `SYNCED`, |
| 75 | + }, |
| 76 | + ]) |
| 77 | + return null |
| 78 | + }, |
| 79 | +} |
| 80 | +``` |
| 81 | + |
| 82 | + |
| 83 | +As these are all functions, you could reuse common shapes across routes e.g. user and org data. You can also flexibly say what “done” is—e.g. for some routes, somewhat stale data is fine, so don’t wait on loading for the freshest data to be loaded from the server. |
| 84 | + |
| 85 | +One big current caveat to all this is that ElectricSQL is still working on shipping Shapes as of December 2023. So currently Samurize syjncs all video data to each client as you can't yet say "only sync videos that I'm looking at". Once that arrives, Samurize will be able to lazily sync data as needed making this approach suitable for however many videos people summarize. |
| 86 | + |
| 87 | +### Pre-running queries |
| 88 | + |
| 89 | +The other problem I ran into is that while local SQLite queries run very fast, they are async and take a bit of time, so on transitioning to a route, there’s some number of frames where nothing renders. So while fast, it causes a very noticeable blink in the UI. |
| 90 | + |
| 91 | +I fixed that by running each routes’ queries in the loader function, storing those in a cache, and then using the pre-run query results until the reactive live queries start running. |
| 92 | + |
| 93 | +The difference is stark: |
| 94 | + |
| 95 | +### Glitchy transition |
| 96 | + |
| 97 | +<video controls width="100%"> |
| 98 | + <source src="./glitchy-route-transition.mp4" type="video/mp4" /> |
| 99 | +</video> |
| 100 | + |
| 101 | +### Smooth transition |
| 102 | + |
| 103 | +<video controls width="100%"> |
| 104 | + <source src="./smooth-route-transition.mp4" type="video/mp4" /> |
| 105 | +</video> |
| 106 | + |
| 107 | + |
| 108 | +It wasn’t much code but I think it’s a critical pattern to follow so I’ll be pulling it out into its own library very soon. |
| 109 | + |
| 110 | +This is similar to how people use tools like [TanStack Query](https://tanstack.com/router/v1/docs/guide/external-data-loading#a-more-realistic-example-using-tanstack-query) — where they prefetch in the loader to warm the cache so the query hook in the route component can respond immediately. |
| 111 | + |
| 112 | + |
| 113 | +```tsx |
| 114 | +import { Route } from '@tanstack/react-router' |
| 115 | + |
| 116 | +const postsQueryOptions = queryOptions({ |
| 117 | + queryKey: 'posts', |
| 118 | + queryFn: () => fetchPosts, |
| 119 | +}) |
| 120 | + |
| 121 | +const postsRoute = new Route({ |
| 122 | + getParentPath: () => rootRoute, |
| 123 | + path: 'posts', |
| 124 | + // Use the `loader` option to ensure that the data is loaded |
| 125 | + loader: () => queryClient.ensureQueryData(postsQueryOptions), |
| 126 | + component: () => { |
| 127 | + // Read the data from the cache and subscribe to updates |
| 128 | + const posts = useSuspenseQuery(postsQueryOptions) |
| 129 | + |
| 130 | + return ( |
| 131 | + <div> |
| 132 | + {posts.map((post) => ( |
| 133 | + <Post key={post.id} post={post} /> |
| 134 | + ))} |
| 135 | + </div> |
| 136 | + ) |
| 137 | + }, |
| 138 | +}) |
| 139 | +``` |
| 140 | + |
| 141 | + |
| 142 | +## LLM/AI Summarization stuff |
| 143 | + |
| 144 | +The AI bits are pretty standard so I won’t spend a lot of time on them. |
| 145 | + |
| 146 | +When a new video is submitted, it’s pushed to a node.js backend which fetches video metadata including the transcript. The transcript is then summarized using the map/reduce technique. Basically I split longer videos into five minute segments, summarize each segment and then summarize together the segments. This handles transcripts that are longer than token limits but also conveniently gives the user both an overall summary plus more detailed summaries about each segment of the video. Often I’ll only read the overall summary but sometimes I want to learn more so open the more detailed summaries. |
| 147 | + |
| 148 | +I started prototyping this with OpenAI’s GPT 3.5 model but got frustrated as response times were highly variable. Sometimes I’d wait as long as 45-60 seconds for a summarization to finish. With that speed, I might as well just skim through the video! So I went looking for other options and found [Perplexity.ai’s new LLM API](https://blog.perplexity.ai/blog/introducing-pplx-api) which hosts a number of open source models. I tested them all and Mistral 7B was very fast (5-10 seconds!) & acceptably good at summarization so switched over. |
| 149 | + |
| 150 | +Even basic LLMs are already good enough at many tasks so I think many workloads will keep shifting to whatever is fastest/cheapest. |
| 151 | + |
| 152 | +## Real-timey bits |
| 153 | + |
| 154 | +This was a pretty stock standard app. The only thing I did which really took advantage of ElectricSQL superpowers was the progress bar for indicating progress of the summarization. |
| 155 | + |
| 156 | +<video controls width="100%"> |
| 157 | + <source src="./progress-bar.mp4" type="video/mp4" /> |
| 158 | +</video> |
| 159 | + |
| 160 | +As ElectricSQL gives you a full-stack reactive data layer for free — it was quite easy. A progress bar is just a value between 0 & 1, i.e., what percentage of the work has been done. The question for this sort of long-running job is how to get that value from the backend doing the work to the frontend for the user to see. Normally it’d be some sort of string and glue hacky setup where the backend pushes events over websockets which the client code then has custom code to intercept and direct to the component through a global data store of some sort. |
| 161 | + |
| 162 | +But none of that here. I have a `youtube_video` table for each video which has a `progress` column. The backend just keeps writing new values to that as summarization calls finish, and the new value is synced directly into the Video component. |
| 163 | + |
| 164 | +This is what the code looks like: |
| 165 | + |
| 166 | +```tsx |
| 167 | +const video = useLiveQuery(db.youtube_videos.liveUnique({ |
| 168 | + select: { |
| 169 | + title: true, |
| 170 | + author_url: true, |
| 171 | + author_name: true, |
| 172 | + progress: true, |
| 173 | + }, |
| 174 | + where: { id }, |
| 175 | +})) |
| 176 | +``` |
| 177 | + |
| 178 | +The component just [checks if `video.progress !== 1` to see if it should display a progress bar or not.](https://github.com/KyleAMathews/samurize/blob/959e9e0c44aa1a5215888411e06eda12938e7f56/src/routes/video.tsx#L119-L141) |
| 179 | + |
| 180 | +There was no custom code to set this up. Just global reactive state. |
| 181 | + |
| 182 | +## Deployment |
| 183 | + |
| 184 | +This was my first time deploying Electric so it took some experimentation and research to decide on a solution. |
| 185 | + |
| 186 | +An ElectricSQL app has three components |
| 187 | + |
| 188 | +* The client — which works the same as any other web/mobile app — I used React/react-router and am hosting on Vercel. |
| 189 | +* Postgres — any of the 1000s of Postgres hosting options (as long as it supports Logical Replication) |
| 190 | +* The Electric sync service — it runs anywhere Docker runs and the Samurize instance is currently using ~325 MB of ram. |
| 191 | + |
| 192 | +Postgres hosting gives you a lot of nice things like backups and a dashboard. But for a side project like this, I don’t care about losing data really, and psql works for a dashboard. So I got a DigitalOcean VM and created a Docker Compose file with containers for Postgres and Electric. I needed a way to terminate SSL for an encrypted websocket connection and after looking at Nginx and Caddy, decided on Caddy as it’s extremely easy to use — you just tell it what domain it’s handling and it automatically gets an SSL cert from Let’s Encrypt. Ngnix works well too but it’s much more manual to get certs. I was pleasantly surprised to see Caddy is only using ~13 MB of ram w/ my extremely low traffic side project. |
| 193 | + |
| 194 | +[Check out the Docker Compose file](https://github.com/KyleAMathews/samurize/blob/main/src/backend/compose/docker-compose-prod.yaml). |
| 195 | + |
| 196 | +The backend Node.js process I run on the same VM with PM2. It’s not actually exposed to the world as it’s just listening for new inserts in the database to do the LLM work. |
0 commit comments