Advanced Concepts in Python — II
An In-depth Exploration of First-Class Functions, Higher-Order Functions, Scopes, and Closures
In the first part of the series, I discussed Iterators, Generators, Coroutines, and Iterator Protocol. The link will be given at the end of the Blog.
In this part, I will go into First-Class Functions, Higher-Order Functions, Scopes, and Closures to lay the foundation for decorators and Context Managers in the next part of the Blog.
Topics to be Covered
- First-Class Functions
- Higher-Order Functions
- NameSpace
- Scopes
- Closures
First-Class Functions
Python treats functions as Objects like lists or dicts etc. This behavior makes functions in python the first-class Object and so it enjoys all the perks of Object.
Functions without parentheses are not executed instantly. It's like a reference pointing to that function and we pass to and return a function from another function this way.
Key features of First-Class Functions
1. Functions in Python can be assigned to a variable.
2. Function can be passed to or returned from another function.
3. Python functions can be loaded inside other data structures such as list, dict, etc.
In cell 2, I am assigning the “add” function to the “fc_func” variable (parenthesis after add isn't used because we are just assigning it and don't want it to execute). Then we can use “fc_func” as the “add” function.
In cell 3, I created a function square and then loaded the function variable name inside a list (10 times). In cell 4, I am iterating through that “func_list” and passing the index (getting it from enumerate) as the argument to the square function.
Higher-Order Functions
The functions that take a function as an argument or return a function are called Higher-Order Functions. Seems a bit confusing with First Class functions? The function that is passed is a First-class function, and the function that takes that function as an argument is a Higher-Order function.
Python has quite several Higher-Order Functions in-built like map, reduce (it comes from functools module), filter. All these three higher-order functions take a function and iterables like list, sets etc.
map() → returns an iterator map object.
functools.reduce() → maps the iterable to a single value after applying the function on two-element at a time. Eg: the sum of a list.
filter() → returns an iterator object for the values for which the function returns true.
In cell 3, I am creating a higher-order function “master_splitter” that takes in a function and a sentence as arguments. In cells 5 and 6, I am passing different functions to the “master_splitter” and it returns results according to the function given.
In the next part, I am creating a function “division_combo” that returns another function according to the argument “isQuotient” passed. If “isQuotient” is true, then it returns a function that calculates the quotient else it returns a function that gives the remainder.
In cell 10, I gave an example of the filter function in Python that returns even numbers.
NameSpace
Before jumping into “Variable Scopes” in Python, let me give you a brief introduction to “Python Namespace”.
Python maintains a mapping of unique names for every method or variable. This system helps python to avoid name collisions and reuse names according to different namespaces. Behind the curtains, it maintains a dictionary through which python looks for valid names, and if not found throws a NameError.
NameError: name 'name of your undefined variable' is not definedScope
According to Python docs, “scope is a textual region of a Python program where a namespace is directly accessible”. Namespaces define which names are available and scope defines which namespace variable is accessible.
Scopes are hierarchical, with access going from inside to outside but not the other way. So, if a variable is not found in a lower level scope, it will automatically go up the hierarchy to find the variable, and if it doesn't find it in the built-in scope (highest level scope), then it will give a NameError. Let’s take a diagram to understand it better.
In the above diagram, we can access variables up the hierarchy from the local → enclosed → global → built-in scope, but we can't access variables inward like from the enclosed to local scope. This rule is called the “LEGB” rule for variable scopes.
Local Scope
The scope inside the body of a particular function is the local scope. The variable residing in the local scope can't be accessed from outside of this scope.
# x, y is in local scope of add_x_y function
def add_x_y():
x = 2
y = 8
return x+y****************************************************x = 3
def print_x():
print(x)print(x)
# This function will output 3
But, in the second function “print_x”, there is no “x” defined in the “local” scope, it will find x, up the hierarchy of scope in the “global” scope, and will print 3.
Enclosed Scope
This is the scope that lies inside the body of the outer function in a nested function.
def outer_func():
x = 1
def inner_func():
x = 2
print(x) # Output-->2
inner_func()
print(x) # Output -->1
outer_func()******************************************************def outer_func():
y = 2
def inner_func():
x = 1
print(x) # Output -->1
print(y) # Output -->2
print(x) # Output -->NameError
In the first part, we can see the local scope prints the local variable value, the enclosed variable is accessible in the enclosed scope.
In the second part, “y” is defined in the enclosed scope but we can access it in “inner_func” i.e in the local scope, but we can’t access “x” inside the “outer_func” as it's defined in the local scope, so we get an error.
Global Scope
Global scope is when a variable is defined in a python file without it being inside a function body.
xg = 9 # Global Scope
def abc():
xg = 10
print(xg)abc() # Output --> 10
print(xg)# Output -->9
“xg” when printing inside the “abc” local scope overriding the value of the global scope, and so its output is 10. But when outside the “abc” function, it prints “9” the global value.
Built-in Scope
These are python built-in variables and methods that can be accessed anywhere in the python file. like print(), len() etc.
Global and Nonlocal Keywords
Global
The “global” keyword is used to modify a global value from a local scope.
global_val = 12
def val_modifier():
global global_val
global_val = 55
print(global_val) # Output -->55print(global_val) # Output --> 12
val_modifier()
print(global_val) # Output --> 55
In the above code, the “global” keyword is modifying the “global_val” from inside the “val_modifier” local scope.
Nonlocal
This is like global but for nested function. So “nonlocal” keyword modifies the variable in the nearest up hierarchy of a nested or multi-nested function.
def parent_func():
eclosed_val = 9
def child_func():
nonlocal eclosed_val
eclosed_val = 18
print(eclosed_val) # Output --> 18
print(eclosed_val) # Output --> 9
child_func()
print(eclosed_val) # Output --> 18parent_func()
In the above code, I am modifying the value of “eclosed_val” from inside the inner function “child_func” using a nonlocal keyword.
Can we not use nonlocal for global scope?
The answer is no. It will give an error, nonlocal is for nested functions only.
def grandfather_func():
gift = "grandfather clock"
def father_func():
gift = "handwatch"
def child_func():
nonlocal gift
print(gift) # Output --> handwatch
gift = "Smartwatch"
print(gift) # Output --> Smartwatch
child_func()
print(gift)
# Output --> Smartwatch We are overrding the nearest variable up the hierarchy
father_func()
print(gift) # Output --> grandfather clockgrandfather_func()
In the above code, when we use the “nonlocal” inside the “child_func” function, it modifies only the “gift” variable of “father_func” as it is nearest in the up hierarchy but the “gift” of “grandfather_func” remains unchanged.
Closures
Wikipedia definition: A technique for implementing lexically scoped name binding in a language with first-class functions.
Before explaining this “not-so-clear” definition, let me review some terms —
a. Nested Function — A inner function inside an outer function.
b. Higher-order Function — A function that takes or returns another function.
c. Enclosing scope — Body of the outer function where the outer function variable and inner function reside.
In python, the inner function remembers and can access the variable in the enclosing scope, even after the outer function's execution is completed. These variables are bound to that inner function returned by that outer function. This is the idea of closure. Lets take an example:
A nested higher-order function “parent_func”, returns another function “child_func”. The variables “m”, “p”, and “x” of the enclosed scope are used by the inner function “child_func”.
Now, I am passing 10 to “parent_func”, then execution gets completed and the inner function “child_func” is stored in “c”. This “c” still has access to these variables “m”, “p”, and “x” of the outer function. We can see that there are those enclosed scope values printed when “c” is called. Simply, it attaches the variables of the enclosed scope to the returned function.
Every function has a “__closure__” attribute to inspect how many closure objects are attached to that function and to get the value of the closures, use the “cell_contents” attribute over the tuple returned by the closure attribute:
c.__closure__[index].cell_contentsClosures are used in decorators, sometimes used in quick implementations instead of defining a whole class when there is only a single method present.
With Closure, we came to the end of this blog. In the next part, I will dive deep into decorators and context managers.
“Life is short (You need Python)” — Bruce Eckel
To check out the first part of the blog “Advanced Concepts in Python — I” click here.
To check out my series on “Statistics for Machine Learning” click here.
