Nelder-Mead Optimization Example in Python

    Optimization is a technique used to find the minimum or maximum value of a function within a set of available options. For instance, finding the shortest path from point A to point B by evaluating multiple alternative directions serves as a simple example of an optimization problem. Generally speaking, optimization is about discovering the most efficient or effective solution to a problem, taking into consideration certain constraints.

    In this tutorial, I'll explain how to use Nelder-Mead method to find a minimum of a given function using minimize() function of SciPy library in Python.

    The tutorial covers:

1. Nelder-Mead method implementation
2. Source code listing
   

 We'll start by loading the required libraries.

from scipy.optimize import minimize 
import matplotlib.pyplot as plt
import numpy as np 

Nelder-Mead method implementation

 

   The Nelder-Mead optimization method is a direct search optimization algorithm used for finding the minimum of an objective function in a multidimensional space. Unlike gradient-based methods, Nelder-Mead doesn't require the calculation of derivatives. It is particularly useful for optimizing objective functions that may be non-differentiable, noisy, or have irregular landscapes. 

    First, we'll define the objective function to find its minimum and visualize it in a graph.

 

def func(x):
    x = x**2+2*np.sin(x*np.pi)
    return x

x = np.arange(-2, 2, 0.01)
y = func(x)

plt.title("Given function in x range")
plt.plot(x, y)
plt.grid()
plt.show() 

 

 

    In Python, the Nelder-Mead optimization method is often implemented using the minimize() function from the SciPy library. We use the minimize() function to find the minimum using the Nelder-Mead method. This function serves as a common interface for implementing various optimization methods. It requires initial starting points, and 'nelder-mead' is set as the method.

 

x0 = -1
result = minimize(func, x0, method="nelder-mead")
print(result) 

final_simplex: (array([[-0.45380859],
       [-0.45390625]]), array([-1.77303645, -1.77303644]))
           fun: -1.7730364463746562
       message: 'Optimization terminated successfully.'
          nfev: 30
           nit: 15
        status: 0
       success: True
             x: array([-0.45380859])

    Here, result contains the following attributes:

    fun - value of objective function 

    nfev -  the number of evaluation, 

    nit - the number of iterations, 

    success - the existence of optimizer, 

    x -  the solution of the optimization,

    status -  termination status,

 We can describe determined minimum in a graph as shown below.

plt.title("Describing the function minimum")
plt.plot(x, y, label="y")
plt.plot(result['x'], result['fun'], 'sr', label="minimum")
plt.legend(loc='best', fancybox=True, shadow=True)
plt.grid()
plt.show()

 

    We can add more starting points and search the minimum again. The method searches new minimum from the given point in a range data. 

x0 = [-1, 1]
ig, ax = plt.subplots(2, figsize=(6, 8))
i = 0
for i in range(len(x0)):
    result = minimize(func,  x0[i], method="nelder-mead")
    ax[i].plot(x, y, label="y")
    ax[i].plot(result['x'], result['fun'], 'sr', label="minimum")
    ax[i].set_title("Starts from " + str(x0[i]))
    ax[i].legend(loc='best', fancybox=True, shadow=True)
    ax[i].grid()
    i = i + 1

plt.tight_layout()        
plt.show() 

 

 

 

    In this tutorial, we've briefly learned how to use Nelder-Mead method with minimize() function in Python. The full source code is listed below. 

 

 

Source code listing

from scipy.optimize import minimize 
import matplotlib.pyplot as plt
import numpy as np

def func(x):
    x = x**2+2*np.sin(x*np.pi)
    return x

x = np.arange(-2, 2, 0.01)
y = func(x)

plt.title("Given function in x range")
plt.plot(x, y)
plt.grid()
plt.show()

x0 = -1
result = minimize(func,  x0, method="nelder-mead")
print(result)

plt.title("Describing the function minimum")
plt.plot(x, y, label="y")
plt.plot(result['x'], result['fun'], 'sr', label="minimum")
plt.legend(loc='best', fancybox=True, shadow=True)
plt.grid()
plt.show()


x0 = [-1, 1]
ig, ax = plt.subplots(2, figsize=(6, 8))
i = 0
for i in range(len(x0)):
    result = minimize(func,  x0[i], method="nelder-mead")
    ax[i].plot(x, y, label="y")
    ax[i].plot(result['x'], result['fun'], 'sr', label="minimum")
    ax[i].set_title("Starts from " + str(x0[i]))
    ax[i].legend(loc='best', fancybox=True, shadow=True)
    ax[i].grid()
    i = i + 1

plt.tight_layout()        
plt.show()

 

 

References:

1. SciPy Nelder-Mead method