Deploy to Github Pages on Sun Jan 7 12:29:51 UTC 2024

Author: zserge

posts/ai-nn/index.html

@@ -101,13 +101,13 @@
 <span class=c1>// Params: 1, 1, -1, 1, 1, 0, -1.8, 0.9, 0
 </span></code></pre></div><p>These params are not even close to the ones we used at the beginning, but nevertheless - the new network predicts a XOR function just fine.</p><p>What about more complex problems? There is a well-known &ldquo;two moon&rdquo; problem, where 2D points that are shaped like two almost-overlapping moons needed to be classified. A network needs to learn how to tell by the point parameters (x,y) which moon does it belong to.</p><p>Test data for the two moons problem can be generated like this:</p><div class=highlight><pre class=chroma><code class=language-js data-lang=js><span class=c1>// Test data generator for the &#34;two moons&#34; problem
 </span><span class=c1></span><span class=kr>const</span> <span class=nx>moons</span> <span class=o>=</span> <span class=p>(</span><span class=nx>n</span><span class=p>)</span> <span class=p>=&gt;</span> <span class=p>{</span>
-	<span class=kr>const</span> <span class=nx>unif</span> <span class=o>=</span> <span class=p>(</span><span class=nx>low</span><span class=p>,</span> <span class=nx>high</span><span class=p>)</span> <span class=p>=&gt;</span> <span class=nb>Math</span><span class=p>.</span><span class=nx>random</span><span class=p>()</span> <span class=o>*</span> <span class=p>(</span><span class=nx>high</span> <span class=o>-</span> <span class=nx>low</span><span class=p>)</span> <span class=o>+</span> <span class=nx>low</span><span class=p>;</span>
+  <span class=kr>const</span> <span class=nx>unif</span> <span class=o>=</span> <span class=p>(</span><span class=nx>low</span><span class=p>,</span> <span class=nx>high</span><span class=p>)</span> <span class=p>=&gt;</span> <span class=nb>Math</span><span class=p>.</span><span class=nx>random</span><span class=p>()</span> <span class=o>*</span> <span class=p>(</span><span class=nx>high</span> <span class=o>-</span> <span class=nx>low</span><span class=p>)</span> <span class=o>+</span> <span class=nx>low</span><span class=p>;</span>
   <span class=kr>const</span> <span class=nx>data</span> <span class=o>=</span> <span class=p>[],</span> <span class=nx>labels</span> <span class=o>=</span> <span class=p>[];</span>
   <span class=k>for</span> <span class=p>(</span><span class=kd>let</span> <span class=nx>i</span> <span class=o>=</span> <span class=mi>0</span><span class=p>;</span> <span class=nx>i</span> <span class=o>&lt;</span> <span class=nb>Math</span><span class=p>.</span><span class=nx>PI</span><span class=p>;</span> <span class=nx>i</span> <span class=o>+=</span> <span class=p>(</span><span class=nb>Math</span><span class=p>.</span><span class=nx>PI</span> <span class=o>*</span> <span class=mi>2</span><span class=p>)</span> <span class=o>/</span> <span class=nx>n</span><span class=p>)</span> <span class=p>{</span>
     <span class=nx>data</span><span class=p>.</span><span class=nx>push</span><span class=p>([</span><span class=nb>Math</span><span class=p>.</span><span class=nx>cos</span><span class=p>(</span><span class=nx>i</span><span class=p>)</span> <span class=o>+</span> <span class=nx>unif</span><span class=p>(</span><span class=o>-</span><span class=mf>0.1</span><span class=p>,</span> <span class=mf>0.1</span><span class=p>)</span> <span class=o>-</span> <span class=mf>0.5</span><span class=p>,</span> <span class=nb>Math</span><span class=p>.</span><span class=nx>sin</span><span class=p>(</span><span class=nx>i</span><span class=p>)</span> <span class=o>+</span> <span class=nx>unif</span><span class=p>(</span><span class=o>-</span><span class=mf>0.1</span><span class=p>,</span> <span class=mf>0.1</span><span class=p>)</span> <span class=o>-</span> <span class=mf>0.3</span><span class=p>]);</span>
     <span class=nx>labels</span><span class=p>.</span><span class=nx>push</span><span class=p>([</span><span class=mi>0</span><span class=p>]);</span>
-		<span class=nx>data</span><span class=p>.</span><span class=nx>push</span><span class=p>([</span><span class=mf>0.5</span> <span class=o>-</span> <span class=nb>Math</span><span class=p>.</span><span class=nx>cos</span><span class=p>(</span><span class=nx>i</span><span class=p>)</span> <span class=o>+</span> <span class=nx>unif</span><span class=p>(</span><span class=o>-</span><span class=mf>0.1</span><span class=p>,</span> <span class=mf>0.1</span><span class=p>),</span> <span class=mf>0.2</span> <span class=o>-</span> <span class=nb>Math</span><span class=p>.</span><span class=nx>sin</span><span class=p>(</span><span class=nx>i</span><span class=p>)</span> <span class=o>+</span> <span class=nx>unif</span><span class=p>(</span><span class=o>-</span><span class=mf>0.1</span><span class=p>,</span> <span class=mf>0.1</span><span class=p>)]);</span>
-		<span class=nx>labels</span><span class=p>.</span><span class=nx>push</span><span class=p>([</span><span class=mi>1</span><span class=p>]);</span>
+  	<span class=nx>data</span><span class=p>.</span><span class=nx>push</span><span class=p>([</span><span class=mf>0.5</span> <span class=o>-</span> <span class=nb>Math</span><span class=p>.</span><span class=nx>cos</span><span class=p>(</span><span class=nx>i</span><span class=p>)</span> <span class=o>+</span> <span class=nx>unif</span><span class=p>(</span><span class=o>-</span><span class=mf>0.1</span><span class=p>,</span> <span class=mf>0.1</span><span class=p>),</span> <span class=mf>0.2</span> <span class=o>-</span> <span class=nb>Math</span><span class=p>.</span><span class=nx>sin</span><span class=p>(</span><span class=nx>i</span><span class=p>)</span> <span class=o>+</span> <span class=nx>unif</span><span class=p>(</span><span class=o>-</span><span class=mf>0.1</span><span class=p>,</span> <span class=mf>0.1</span><span class=p>)]);</span>
+  	<span class=nx>labels</span><span class=p>.</span><span class=nx>push</span><span class=p>([</span><span class=mi>1</span><span class=p>]);</span>
   <span class=p>}</span>
   <span class=k>return</span> <span class=p>[</span><span class=nx>data</span><span class=p>,</span> <span class=nx>labels</span><span class=p>];</span>
 <span class=p>}</span>
@@ -116,8 +116,8 @@
 </code></pre></div><p>We can now define the structure of our network and train it:</p><div class=highlight><pre class=chroma><code class=language-js data-lang=js><span class=kr>const</span> <span class=nx>nn</span> <span class=o>=</span> <span class=nx>MLP</span><span class=p>([</span><span class=mi>2</span><span class=p>,</span><span class=mi>8</span><span class=p>,</span><span class=mi>4</span><span class=p>,</span><span class=mi>1</span><span class=p>])</span>
 
 <span class=k>for</span> <span class=p>(</span><span class=kd>let</span> <span class=nx>i</span> <span class=o>=</span> <span class=mi>0</span><span class=p>;</span> <span class=nx>i</span> <span class=o>&lt;</span> <span class=mi>100</span><span class=p>;</span> <span class=nx>i</span><span class=o>++</span><span class=p>)</span> <span class=p>{</span>
-	<span class=kr>const</span> <span class=nx>e</span> <span class=o>=</span> <span class=nx>nn</span><span class=p>.</span><span class=nx>loss</span><span class=p>(</span><span class=nx>X</span><span class=p>,</span> <span class=nx>Y</span><span class=p>,</span> <span class=mf>0.2</span><span class=p>);</span>
-	<span class=nx>console</span><span class=p>.</span><span class=nx>log</span><span class=p>(</span><span class=nx>i</span><span class=p>,</span> <span class=nx>e</span><span class=p>.</span><span class=nx>val</span><span class=p>);</span>
+  <span class=kr>const</span> <span class=nx>e</span> <span class=o>=</span> <span class=nx>nn</span><span class=p>.</span><span class=nx>loss</span><span class=p>(</span><span class=nx>X</span><span class=p>,</span> <span class=nx>Y</span><span class=p>,</span> <span class=mf>0.2</span><span class=p>);</span>
+  <span class=nx>console</span><span class=p>.</span><span class=nx>log</span><span class=p>(</span><span class=nx>i</span><span class=p>,</span> <span class=nx>e</span><span class=p>.</span><span class=nx>val</span><span class=p>);</span>
 <span class=p>}</span>
 </code></pre></div><p>In the first few iterations the error drops rapidly to only ~5% and then the network keeps reducing the error slowly and steadily. Finally, after 100 iterations it manages to classify the points belonging to the two moons perfectly:</p><p><img src=/images/nn/moons.png alt=moons></p><p>Full code is available on <a href=https://github.com/zserge/aint/tree/main/nn>GitHub</a>.</p><p>Of course training a real practical network is much harder. It&rsquo;s a very slow process, that would probably require a GPU or a <a href=https://en.wikipedia.org/wiki/Tensor_Processing_Unit>TPU</a>. You would need a proper framework like PyTorch or Tensorflow or Keras (although even a <a href=https://github.com/tinygrad/tinygrad>tinygrad</a> works surprisingly well for small cases and is so much easier to learn and understand). You would have to be aware of proper weights initialisation, optimisers, managing learning rate, regulations, dropout layers, batching and many other topics that turn machine learning into its own branch of science.</p><h2 id=ai>AI?</h2><p>But even our toy neural network despite its slowness is a real machine leaning algorithm. Is it intelligent though? It can&rsquo;t speak yet, but it can be trained automatically and gain knowledge about the problems it never seen before. What if instead of numbers such a network could take words as an input and produce sentences as an output?</p><p>Next part: <a href=/posts/ai-llm>Large Language Models</a>.</p><p>I hope you’ve enjoyed this article. You can follow – and contribute to – on <a href=https://github.com/zserge>Github</a>, <a href=https://mastodon.social/@zserge>Mastodon</a>, <a href=https://twitter.com/zsergo>Twitter</a> or subscribe via <a href=/rss.xml>rss</a>.</p><p class=date style=text-align:right><em>Jan 03, 2024</em></p><p>See also:
 <a href=/posts/ai-markov/>AI or ain't: Markov Chains</a> and <a href=/posts/>more</a>.</p></div><footer><p>&copy;2012&ndash;2023 &#183;