# HG changeset patch # User Paul Boddie # Date 1341707187 -7200 # Node ID 1f42e393ed152674d47a94eabb8aafdf6b20312f # Parent c7d9132bd609cdfdc14ae39fcc1c9ca8f3f09118 Added circular import detection by separating module loading from module "completion" and processing, thus preventing attribute usage observations from being made before processing of an affected module's code can begin. Simplified module loading in the importer somewhat. diff -r c7d9132bd609 -r 1f42e393ed15 micropython/__init__.py --- a/micropython/__init__.py Sat Jul 07 20:40:09 2012 +0200 +++ b/micropython/__init__.py Sun Jul 08 02:26:27 2012 +0200 @@ -94,6 +94,8 @@ "Finalise the program." + self.importer.complete_modules() + # Need the tables to finalise. objtable = self.get_object_table() @@ -416,6 +418,10 @@ self.importers = {} + # Modules involved in circular imports. + + self.circular_imports = set() + # Constant records. self.constant_values = {} @@ -440,6 +446,7 @@ # Status information. + self.completed = 0 self.vacuumed = 0 self.finalised = 0 @@ -455,6 +462,18 @@ return self.modules[name] + def complete_modules(self): + + "Complete the processing of modules." + + if self.completed: + return + + self.get_module("__builtins__").complete() + self.get_module("__main__").complete() + + self.completed = 1 + # General maintenance. def vacuum(self, objtable): @@ -968,58 +987,43 @@ # the search path. path = name.split(".") - m = self.find_in_path(path[0]) - if not m: - if self.verbose: - print >>sys.stderr, "Not found (%s)" % path[0] - return None # NOTE: Import error. - d, filename = m + path_so_far = [] + top = module = None - # Either acquire a reference to an already-imported module, or load the - # module from a file. + for p in path: - top = module = self.load_from_file(filename, path[0], importer) - - # For hierarchical names, traverse each path component... + # Get the module's filesystem details. - if len(path) > 1: - if not d: - if self.verbose: - print >>sys.stderr, "No package (%s)" % filename - return None # NOTE: Import error (package not found). + if not path_so_far: + m = self.find_in_path(p) else: - self.add_submodules(d, module) + m = self.find(d, p) - path_so_far = path[:1] - for p in path[1:]: - path_so_far.append(p) + if not m: + if self.verbose: + print >>sys.stderr, "Not found (%s)" % p + return None # NOTE: Import error. - # Find the package or module concerned. + # Get the module itself. - m = self.find(d, p) - if not m: - if self.verbose: - print >>sys.stderr, "Not found (%s)" % p - return None # NOTE: Import error. - d, filename = m - module_name = ".".join(path_so_far) + d, filename = m + path_so_far.append(p) + module_name = ".".join(path_so_far) + submodule = self.load_from_file(filename, module_name, importer) - # Either reference an imported module or load one from a file. - - submodule = self.load_from_file(filename, module_name, importer) - - if d: - self.add_submodules(d, module) - + if module is None: + top = submodule + else: # Store the submodule within its parent module. module.set_module(p, submodule) - module = submodule + + module = submodule - # Stop descending if no package was found. + # Stop descending if no package was found. - if not d: - break + if not d: + break # Return either the deepest or the uppermost module. diff -r c7d9132bd609 -r 1f42e393ed15 micropython/data.py --- a/micropython/data.py Sat Jul 07 20:40:09 2012 +0200 +++ b/micropython/data.py Sun Jul 08 02:26:27 2012 +0200 @@ -2023,6 +2023,11 @@ self.all_objects = set() + # Whether the module is imported in a circular fashion, exposing the + # unfinished namespace to external modification. + + self.circular_import = self in importer.circular_imports + # A set of global names that cannot support combined attribute usage # observations because they may be modified within functions during # initialisation. @@ -2084,29 +2089,46 @@ if not self.namespace.has_key(name): self.namespace[name] = Attr(None, self, name) + # Indicate that there is at least one assignment to the name, although + # no actual values are recorded. + attr = self.namespace[name] attr.update_assignments(1, False) + def set(self, name, value, single_assignment=1): + + """ + Set the module attribute with the given 'name', having the given 'value' + that is assigned once if 'single_assignment' is omitted to given as a + true value. + + Where the module is imported before it is completely initialised, all + assignments will be regarded as multiple assignments since it is + possible that an external assignment to the name may occur. + """ + + NamespaceDict.set(self, name, value, not self.circular_import and single_assignment) + # Attribute usage methods that apply to module globals in certain # circumstances. def _use_attribute(self, name, attrname, value=None): - if name not in self.modified_names: + if name not in self.modified_names and not self.circular_import: return NamespaceDict._use_attribute(self, name, attrname, value) else: self.importer.use_name(attrname, self.full_name(), value) return [] def _define_attribute_user_for_name(self, node, name): - if name not in self.modified_names: + if name not in self.modified_names and not self.circular_import: NamespaceDict._define_attribute_user_for_name(self, node, name) def _init_attribute_user_for_name(self, node, name): - if name not in self.modified_names: + if name not in self.modified_names and not self.circular_import: NamespaceDict._init_attribute_user_for_name(self, node, name) def _define_attribute_accessor(self, name, attrname, node, value): - if name not in self.modified_names: + if name not in self.modified_names and not self.circular_import: NamespaceDict._define_attribute_accessor(self, name, attrname, node, value) else: self.importer.use_name(attrname, self.full_name(), value) diff -r c7d9132bd609 -r 1f42e393ed15 micropython/inspect.py --- a/micropython/inspect.py Sat Jul 07 20:40:09 2012 +0200 +++ b/micropython/inspect.py Sun Jul 08 02:26:27 2012 +0200 @@ -90,8 +90,9 @@ order: 1. parse - 2. vacuum - 3. finalise + 2. process + 3. vacuum + 4. finalise A module importer can be expected to perform these invocations. """ @@ -111,6 +112,7 @@ self.optimisations = self.importer.optimisations self.builtins = self.importer.modules.get("__builtins__") self.loaded = 0 + self.completed = 0 # Current expression state. @@ -129,28 +131,33 @@ "Parse the file having the given 'filename'." - module = compiler.parseFile(filename) - self.process(module) + self.astnode = module = compiler.parseFile(filename) + + # Detect and record imports and globals declared in the module. + + self.process_structure(module) - def process(self, module): + def complete(self): + if not self.completed: + self.completed = 1 + self.process() + if self.importer.verbose: + print >>sys.stderr, "Completed import of", self.full_name() + + def process(self): + return self.process_module(self.astnode) + + def process_module(self, module): """ - Process the given 'module', visiting only module-level code and only - extracting global declarations from functions. In order to support - deferred imports, the function code is processed separately after all - modules have been initially processed. + Process the given 'module', visiting module-level code and function + code. """ - self.astnode = module - # Add __name__ to the namespace. self.store("__name__", self._visitConst(self.full_name())) - # Detect and record globals declared in the module. - - self.process_globals(module) - # Visit module-level code, also recording global names. processed = self.dispatch(module) @@ -187,14 +194,18 @@ self._visitFunctionBody(node, namespaces) namespaces[-1].finalise_attribute_usage() - def process_globals(self, node): + def process_structure(self, node): """ Within the given 'node', process global declarations, adjusting the - module namespace. + module namespace, and import statements, building a module dependency + hierarchy. """ for n in node.getChildNodes(): + + # Module global detection. + if isinstance(n, compiler.ast.Global): for name in n.names: @@ -203,10 +214,95 @@ # just indicate that the name cannot be relied upon for # various observations. - module.modify_name(name) + self.modify_name(name) + + # Module import declarations. + + elif isinstance(n, compiler.ast.From): + + # Load the mentioned module. + + module = self.importer.load(n.modname, 1, importer=n) + self.record_import(n.modname, n) + + # Speculatively load modules for names beneath the module. + + for name, alias in n.names: + modname = n.modname + "." + name + self.record_import(modname, n) + + elif isinstance(n, compiler.ast.Import): + + # Load the mentioned module. + + for name, alias in n.names: + self.record_import(name, n) + + # Nodes using operator module functions. + + elif operator_functions.has_key(n.__class__.__name__) or \ + isinstance(n, (compiler.ast.AugAssign, compiler.ast.Compare)): + + n._module = self.importer.load("operator") else: - self.process_globals(n) + self.process_structure(n) + + def get_module_paths(self, name): + + """ + Return the paths of modules leading to the module having the given + 'name'. + """ + + names = [] + parts = [] + for part in name.split("."): + parts.append(part) + names.append(".".join(parts)) + return names + + def record_import(self, name, node): + + """ + Record an import of a module with the given 'name' occurring at the + given 'node'. + """ + + module = self.importer.load(name, 1, importer=node) + if module and not module.loaded: + self.importer.circular_imports.add(module) + + def complete_import(self, name, return_leaf): + + """ + Complete the import of the module with the given 'name', returning the + module itself if 'return_leaf' is a true value, or returning the root of + the module hierarchy if 'return_leaf' is a false value. + """ + + top = module = None + + for modname in self.get_module_paths(name): + + # Attempt to get the module, returning None for non-existent + # modules. + + try: + module = self.importer.get_module(modname) + except KeyError: + return None + + if module: + module.complete() + + if top is None: + top = module + + if return_leaf: + return module + else: + return top def vacuum(self): @@ -531,10 +627,11 @@ # Generic support for classes of operations. - def _ensureOperators(self): + def _ensureOperators(self, node): attr, scope, namespace = self._get_with_scope("$operator") if attr is None: - module = self.importer.load("operator") + module = node._module + module.complete() self["$operator"] = module else: module = attr.get_value() @@ -544,7 +641,7 @@ "Accounting method for the operator 'node'." - operator_module = self._ensureOperators() + operator_module = self._ensureOperators(node) operator_fn = operator_functions[operator_name or node.__class__.__name__] self.use_specific_attribute(operator_module.full_name(), operator_fn) return self.OP(node) @@ -795,7 +892,7 @@ # Accounting. operator_fn = operator_functions.get(node.op) - operator_module = self._ensureOperators() + operator_module = self._ensureOperators(node) self.use_specific_attribute(operator_module.full_name(), operator_fn) # Process the assignment. @@ -920,7 +1017,7 @@ # For operators, reference the specific function involved. if operator_fn is not None: - operator_module = self._ensureOperators() + operator_module = self._ensureOperators(node) self.use_specific_attribute(operator_module.full_name(), operator_fn) # Define __contains__ usage on the next node. @@ -1003,12 +1100,7 @@ self.resume_broken_branches() def visitFrom(self, node): - module = self.importer.load(node.modname, 1, importer=node) - if module and not module.loaded: - print >>sys.stderr, "Warning: a circular import of %s is being attempted in %s" % (node.modname, self.full_name()) - - if module is None and self.importer.verbose: - print >>sys.stderr, "Warning:", node.modname, "not imported." + module = self.complete_import(node.modname, True) for name, alias in node.names: @@ -1020,13 +1112,9 @@ # Missing names may refer to submodules. - if not module.has_key(name): - submodule = self.importer.load(node.modname + "." + name, 1, importer=node) - if submodule: - if not submodule.loaded: - print >>sys.stderr, "Warning: a circular import of %s.%s is being attempted in %s" % ( - node.modname, name, self.full_name()) - + submodule = self.complete_import(node.modname + "." + name, True) + if submodule: + if not module.has_key(name): module.store(name, submodule) # Complete the import if the name was found. @@ -1035,17 +1123,6 @@ attr = module[name] self.store(alias or name, attr) self.use_specific_attribute(module.full_name(), name) - if isinstance(attr.get_value(), Module) and not attr.get_value().loaded: - submodule = self.importer.load(attr.get_value().name, importer=node) - - # For circular imports, invalidate attribute usage for - # all exported names of submodules whose names are - # specified in the from statement. - - if submodule and not submodule.loaded: - for n in submodule.keys(): - submodule.modify_name(n) - continue # Support the import of names from missing modules. @@ -1061,16 +1138,6 @@ attr = module[n] self.store(n, attr) self.use_specific_attribute(module.full_name(), n) - if isinstance(attr.get_value(), Module) and not attr.get_value().loaded: - submodule = self.importer.load(attr.get_value().name, importer=node) - - # For circular imports, invalidate attribute usage - # for all exported names of submodules provided by - # the module. - - if submodule and not submodule.loaded: - for subn in submodule.keys(): - submodule.modify_name(subn) def visitFunction(self, node): return self._visitFunction(node, node.name) @@ -1141,21 +1208,12 @@ def visitImport(self, node): for name, alias in node.names: + module = self.complete_import(name, alias) if alias is not None: - module = self.importer.load(name, 1, importer=node) or UnresolvedName(None, name, self) - self.store(alias, module) + self.store(alias, module or UnresolvedName(None, name, self)) else: - module = self.importer.load(name, importer=node) or UnresolvedName(None, name.split(".")[0], self) - self.store(name.split(".")[0], module) - - circular = module and not module.loaded - - # For circular imports, invalidate attribute usage for all exported - # names of modules. - - if module and not module.loaded: - for n in module.keys(): - module.modify_name(n) + name_used = name.split(".")[0] + self.store(name_used, module or UnresolvedName(None, name_used, self)) visitInvert = _visitOperator diff -r c7d9132bd609 -r 1f42e393ed15 micropython/trans.py --- a/micropython/trans.py Sat Jul 07 20:40:09 2012 +0200 +++ b/micropython/trans.py Sun Jul 08 02:26:27 2012 +0200 @@ -1019,7 +1019,7 @@ else: # NOTE: This may happen because a class attribute is optimised away. - print "Program unit uses unknown name %r." % name + print >>sys.stderr, "Program unit uses unknown name %r." % name def _visitUnary(self, node):