Multi-context execution (py.Context)

.gitignore

@@ -8,3 +8,6 @@ __pycache__
 cover.out
 /junk
 /dist
+
+examples/embedding/embedding
+builtin/testfile

.vscode/launch.json

@@ -0,0 +1,26 @@
+{
+    // Use IntelliSense to learn about possible attributes.
+    // Hover to view descriptions of existing attributes.
+    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
+    "version": "0.2.0",
+    "configurations": [
+        {
+            "name": "Launch file",
+            "type": "go",
+            "request": "launch",
+            "mode": "debug",
+            "program": "${file}"
+        },
+        {
+            "name": "examples/embedding",
+            "type": "go",
+            "request": "launch",
+            "mode": "auto",
+            "program": "${workspaceFolder}/examples/embedding",
+            "cwd": "${workspaceFolder}/examples/embedding",
+            "args": [
+                "mylib-demo.py"
+            ],
+        },
+    ]
+}

README.md

@@ -5,18 +5,20 @@
 [![GoDoc](https://godoc.org/github.com/go-python/gpython?status.svg)](https://godoc.org/github.com/go-python/gpython)
 [![License](https://img.shields.io/badge/License-BSD--3-blue.svg)](https://github.com/go-python/gpython/blob/master/LICENSE)
 
-gpython is a part re-implementation / part port of the Python 3.4
-interpreter to the Go language, "batteries not included".
-
-It includes:
-
-  * runtime - using compatible byte code to python3.4
-  * lexer
-  * parser
-  * compiler
+gpython is a part re-implementation, part port of the Python 3.4
+interpreter in Go.  Although there are many areas of improvement,
+it stands as an noteworthy achievement in capability and potential.
+
+gpython includes:
+
+  * lexer, parser, and compiler
+  * runtime and high-level convenience functions
+  * multi-context interpreter instancing
+  * easy embedding into your Go application
   * interactive mode (REPL) ([try online!](https://gpython.org))
 
-It does not include very many python modules as many of the core
+
+gpython does not include many python modules as many of the core
 modules are written in C not python.  The converted modules are:
 
   * builtins
@@ -27,53 +29,52 @@ modules are written in C not python.  The converted modules are:
 
 ## Install
 
-Gpython is a Go program and comes as a single binary file.
-
-Download the relevant binary from here: https://github.com/go-python/gpython/releases
+Download directly from the [releases page](https://github.com/go-python/gpython/releases) 
 
-Or alternatively if you have Go installed use
+Or if you have Go installed:
 
-    go get github.com/go-python/gpython
-
-and this will build the binary in `$GOPATH/bin`.  You can then modify
-the source and submit patches.
+    go install github.com/go-python/gpython
 
 ## Objectives
 
-Gpython was written as a learning experiment to investigate how hard
+gpython started as an experiment to investigate how hard
 porting Python to Go might be.  It turns out that all those C modules
-are a significant barrier to making a fully functional port.
+are a significant barrier to making gpython a complete replacement
+to CPython.  
 
-## Status
+However, to those who want to embed a highly popular and known language
+into their Go application, gpython could be a great choice over less
+capable (or lesser known) alternatives.
 
-The project works well enough to parse all the code in the python 3.4
-distribution and to compile and run python 3 programs which don't
-depend on a module gpython doesn't support.
+## Status
 
-See the examples directory for some python programs which run with
-gpython.
+gpython currently:
+ - Parses all the code in the Python 3.4 distribution
+ - Runs Python 3 for the modules that are currently supported
+ - Supports concurrent multi-interpreter execution ("multi-context") 
 
 Speed hasn't been a goal of the conversions however it runs pystone at
-about 20% of the speed of cpython.  The pi test runs quicker under
-gpython as I think the Go long integer primitives are faster than the
+about 20% of the speed of CPython.  The [π test](https://github.com/go-python/gpython/tree/master/examples/pi_chudnovsky_bs.py) runs quicker under
+gpython as the Go long integer primitives are likely faster than the
 Python ones.
 
-There are many directions this project could go in.  I think the most
-profitable would be to re-use the
-[grumpy](https://github.com/grumpyhome/grumpy) runtime (which would mean
-changing the object model).  This would give access to the C modules
-that need to be ported and would give grumpy access to a compiler and
-interpreter (gpython does support `eval` for instance).
+@ncw started gpython it in 2013 and work on is sporadic. If you or someone
+you know would be interested to take it futher, it would be much appreciated.
+
+## Getting Started
 
-I (@ncw) haven't had much time to work on gpython (I started it in
-2013 and have worked on it very sporadically) so someone who wants to
-take it in the next direction would be much appreciated.
+The [embedding example](https://github.com/go-python/gpython/tree/master/examples/embedding) demonstrates how to 
+easily embed and invoke gpython from any Go application.
 
-## Limitations and Bugs
+Importantly, gpython is able to run multiple interpreter instances simultaneously,
+allowing you to embed gpython naturally into your Go application.  This makes it
+possible to use gpython in a server situation where complete interpreter 
+independence is an absolute requirement.  See this in action in the [multi-context example](https://github.com/go-python/gpython/tree/master/examples/multi-context)
+
+If you are looking to get involved, a light and easy place to start is adding more convenience functions to [py/util.go](https://github.com/go-python/gpython/tree/master/py/util.go).  See [notes.txt](https://github.com/go-python/gpython/blob/master/notes.txt) for bigger ideas. 
 
-Lots!
 
-## Similar projects
+## Other Projects of Interest
 
   * [grumpy](https://github.com/grumpyhome/grumpy) - a python to go transpiler
 
@@ -86,5 +87,5 @@ or on the [Gophers Slack](https://gophers.slack.com/) in the `#go-python` channe
 ## License
 
 This is licensed under the MIT licence, however it contains code which
-was ported fairly directly directly from the cpython source code under
+was ported fairly directly directly from the CPython source code under
 the [PSF LICENSE](https://github.com/python/cpython/blob/master/LICENSE).

builtin/builtin.go

@@ -162,7 +162,17 @@ func init() {
 		"Warning":                   py.Warning,
 		"ZeroDivisionError":         py.ZeroDivisionError,
 	}
-	py.NewModule("builtins", builtin_doc, methods, globals)
+
+	py.RegisterModule(&py.ModuleImpl{
+		Info: py.ModuleInfo{
+			Name:  "builtins",
+			Doc:   builtin_doc,
+			Flags: py.ShareModule,
+		},
+		Methods: methods,
+		Globals: globals,
+	})
+
 }
 
 const print_doc = `print(value, ..., sep=' ', end='\\n', file=sys.stdout, flush=False)
@@ -178,18 +188,22 @@ func builtin_print(self py.Object, args py.Tuple, kwargs py.StringDict) (py.Obje
 	var (
 		sepObj py.Object = py.String(" ")
 		endObj py.Object = py.String("\n")
-		file   py.Object = py.MustGetModule("sys").Globals["stdout"]
 		flush  py.Object
 	)
+	sysModule, err := self.(*py.Module).Context.GetModule("sys")
+	if err != nil {
+		return nil, err
+	}
+	stdout := sysModule.Globals["stdout"]
 	kwlist := []string{"sep", "end", "file", "flush"}
-	err := py.ParseTupleAndKeywords(nil, kwargs, "|ssOO:print", kwlist, &sepObj, &endObj, &file, &flush)
+	err = py.ParseTupleAndKeywords(nil, kwargs, "|ssOO:print", kwlist, &sepObj, &endObj, &stdout, &flush)
 	if err != nil {
 		return nil, err
 	}
 	sep := sepObj.(py.String)
 	end := endObj.(py.String)
 
-	write, err := py.GetAttrString(file, "write")
+	write, err := py.GetAttrString(stdout, "write")
 	if err != nil {
 		return nil, err
 	}
@@ -219,7 +233,7 @@ func builtin_print(self py.Object, args py.Tuple, kwargs py.StringDict) (py.Obje
 	}
 
 	if shouldFlush, _ := py.MakeBool(flush); shouldFlush == py.True {
-		fflush, err := py.GetAttrString(file, "flush")
+		fflush, err := py.GetAttrString(stdout, "flush")
 		if err == nil {
 			return py.Call(fflush, nil, nil)
 		}
@@ -449,7 +463,7 @@ func builtin___build_class__(self py.Object, args py.Tuple, kwargs py.StringDict
 	}
 	// fmt.Printf("Calling %v with %v and %v\n", fn.Name, fn.Globals, ns)
 	// fmt.Printf("Code = %#v\n", fn.Code)
-	cell, err = py.VmRun(fn.Globals, ns, fn.Code, fn.Closure)
+	cell, err = fn.Context.RunCode(fn.Code, fn.Globals, ns, fn.Closure)
 	if err != nil {
 		return nil, err
 	}
@@ -749,9 +763,9 @@ func builtin_compile(self py.Object, args py.Tuple, kwargs py.StringDict) (py.Ob
 		return nil, py.ExceptionNewf(py.ValueError, "compile(): invalid optimize value")
 	}
 
-	if dont_inherit.(py.Int) != 0 {
-		// PyEval_MergeCompilerFlags(&cf)
-	}
+	// if dont_inherit.(py.Int) != 0 {
+	// 	PyEval_MergeCompilerFlags(&cf)
+	// }
 
 	// switch string(startstr.(py.String)) {
 	// case "exec":
@@ -782,7 +796,7 @@ func builtin_compile(self py.Object, args py.Tuple, kwargs py.StringDict) (py.Ob
 		return nil, err
 	}
 	// result = py.CompileStringExFlags(str, filename, start[mode], &cf, optimize)
-	result, err = compile.Compile(str, string(filename.(py.String)), string(startstr.(py.String)), int(supplied_flags.(py.Int)), dont_inherit.(py.Int) != 0)
+	result, err = compile.Compile(str, string(filename.(py.String)), py.CompileMode(startstr.(py.String)), int(supplied_flags.(py.Int)), dont_inherit.(py.Int) != 0)
 	if err != nil {
 		return nil, err
 	}
@@ -882,9 +896,8 @@ or ... etc.
 `
 
 func isinstance(obj py.Object, classOrTuple py.Object) (py.Bool, error) {
-	switch classOrTuple.(type) {
+	switch class_tuple := classOrTuple.(type) {
 	case py.Tuple:
-		var class_tuple = classOrTuple.(py.Tuple)
 		for idx := range class_tuple {
 			res, _ := isinstance(obj, class_tuple[idx])
 			if res {

builtin/tests/builtin.py

@@ -299,40 +299,43 @@ def gen2():
 doc="print"
 ok = False
 try:
-    print("hello", sep=1)
+    print("hello", sep=1, end="!")
 except TypeError as e:
-    #if e.args[0] != "sep must be None or a string, not int":
-    #   raise
+    if e.args[0] != "print() argument 1 must be str, not int":
+        raise
     ok = True
 assert ok, "TypeError not raised"
 
 try:
-    print("hello", sep=" ", end=1)
+    print("hello", sep=",", end=1)
 except TypeError as e:
-    #if e.args[0] != "end must be None or a string, not int":
-    #   raise
+    if e.args[0] != "print() argument 2 must be str, not int":
+      raise
     ok = True
 assert ok, "TypeError not raised"
 
 try:
-    print("hello", sep=" ", end="\n", file=1)
+    print("hello", sep=",", end="!", file=1)
 except AttributeError as e:
-    #if e.args[0] != "'int' object has no attribute 'write'":
-    #   raise
+    if e.args[0] != "'int' has no attribute 'write'":
+       raise
     ok = True
 assert ok, "AttributeError not raised"
 
 with open("testfile", "w") as f:
-    print("hello", "world", sep=" ", end="\n", file=f)
+    print("hello", "world", end="!\n", file=f, sep=", ")
+    print("hells", "bells",  end="...", file=f)
+    print(" ~", "Brother ", "Foo", "bar", file=f, end="", sep="")
 
 with open("testfile", "r") as f:
-    assert f.read() == "hello world\n"
+    assert f.read() == "hello, world!\nhells bells... ~Brother Foobar"
 
 with open("testfile", "w") as f:
-    print(1,2,3,sep=",",end=",\n", file=f)
+    print(1,2,3,sep=",", flush=False, end=",\n", file=f)
+    print("4",5, file=f, end="!", flush=True, sep=",")
 
 with open("testfile", "r") as f:
-    assert f.read() == "1,2,3,\n"
+    assert f.read() == "1,2,3,\n4,5!"
 
 doc="round"
 assert round(1.1) == 1.0

compile/compile.go

@@ -89,33 +89,36 @@ func init() {
 	py.Compile = Compile
 }
 
-// Compile(source, filename, mode, flags, dont_inherit) -> code object
+// Compile(src, srcDesc, compileMode, flags, dont_inherit) -> code object
 //
 // Compile the source string (a Python module, statement or expression)
-// into a code object that can be executed by exec() or eval().
-// The filename will be used for run-time error messages.
-// The mode must be 'exec' to compile a module, 'single' to compile a
-// single (interactive) statement, or 'eval' to compile an expression.
+// into a code object that can be executed.
+//
+// srcDesc is used for run-time error messages and is typically a file system pathname,
+//
+// See py.CompileMode for compile mode options.
+//
 // The flags argument, if present, controls which future statements influence
 // the compilation of the code.
+//
 // The dont_inherit argument, if non-zero, stops the compilation inheriting
 // the effects of any future statements in effect in the code calling
 // compile; if absent or zero these statements do influence the compilation,
 // in addition to any features explicitly specified.
-func Compile(str, filename, mode string, futureFlags int, dont_inherit bool) (py.Object, error) {
+func Compile(src, srcDesc string, mode py.CompileMode, futureFlags int, dont_inherit bool) (*py.Code, error) {
 	// Parse Ast
-	Ast, err := parser.ParseString(str, mode)
+	Ast, err := parser.ParseString(src, mode)
 	if err != nil {
 		return nil, err
 	}
 	// Make symbol table
-	SymTable, err := symtable.NewSymTable(Ast, filename)
+	SymTable, err := symtable.NewSymTable(Ast, srcDesc)
 	if err != nil {
 		return nil, err
 	}
 	c := newCompiler(nil, compilerScopeModule)
-	c.Filename = filename
-	err = c.compileAst(Ast, filename, futureFlags, dont_inherit, SymTable)
+	c.Filename = srcDesc
+	err = c.compileAst(Ast, srcDesc, futureFlags, dont_inherit, SymTable)
 	if err != nil {
 		return nil, err
 	}
@@ -1342,7 +1345,6 @@ func (c *compiler) NameOp(name string, ctx ast.ExprContext) {
 		default:
 			panic("NameOp: ctx invalid for name variable")
 		}
-		break
 	}
 	if op == 0 {
 		panic("NameOp: Op not set")

compile/compile_data_test.go

@@ -12,7 +12,7 @@ import (
 
 var compileTestData = []struct {
 	in            string
-	mode          string // exec, eval or single
+	mode          py.CompileMode
 	out           *py.Code
 	exceptionType *py.Type
 	errString     string