The Elixir and Python ecosystems each have unique strengths: Elixir excels at building concurrent, fault-tolerant systems, while Python dominates in machine learning, data science, and scientific computing. But what if you could combine both in the same application without the complexity of microservices or external APIs?

Enter Pythonx – a library that embeds a Python interpreter directly into your Elixir application, enabling you to evaluate Python code, call Python libraries, and seamlessly convert data between both languages.

In this post, we’ll explore what Pythonx is, how it works, practical use cases, and when it makes sense to use this powerful interoperability tool.

The Integration Challenge

Before Pythonx, integrating Python with Elixir typically meant choosing between several approaches:

• Ports: Spawning external Python processes and communicating via stdin/stdout
• ErlPort: More sophisticated port-based communication with term serialization
• HTTP APIs: Running Python services separately and calling them over HTTP
• Message queues: Using RabbitMQ or similar for async communication

While these approaches work, they add operational complexity, latency, and require managing separate processes or services. For many use cases, especially in development, prototyping, or when using Livebook for data science workflows, a simpler solution is desirable.

What is Pythonx?

Pythonx is an open-source library maintained by the Livebook team that embeds a Python interpreter directly into your Elixir application using Erlang NIFs (Native Implemented Functions). This means Python runs in the same OS process as your BEAM application, enabling fast, efficient communication between Elixir and Python code.

Key Features

1. In-Process Execution: No external processes or network calls required
2. Automatic Data Conversion: Seamless translation between Python and Elixir data types
3. Modern Package Management: Uses uv for fast, reliable Python dependency management
4. Configurable: Specify Python version and packages at compile time
5. Development-Friendly: Includes a ~PY sigil for interactive use in IEx and Livebook

How Pythonx Works

Under the hood, Pythonx uses NIFs to link against CPython’s shared library and embed the interpreter directly in the BEAM. This architecture provides:

• Direct memory access: No serialization overhead for simple data types
• Low latency: Function calls don’t cross process boundaries
• Shared state: Python globals persist between calls

However, this design also comes with important trade-offs we’ll discuss later.

Installation and Setup

For Mix Scripts

Mix.install([
  {:pythonx, "~> 0.4.0"}
])

For Applications

Add to your mix.exs:

def deps do
  [
    {:pythonx, "~> 0.4.0"}
  ]
end

Then configure in config/config.exs:

config :pythonx,
  python_version: "3.12",
  python_packages: [
    "numpy==1.26.0",
    "pandas==2.1.0",
    "scikit-learn==1.3.0"
  ]

This configuration downloads and bundles Python dependencies at compile time, making them part of your Elixir release.

Basic Usage

Simple Python Evaluation

# Evaluate Python code
Pythonx.eval("2 + 2")
# => 4

# Use variables
Pythonx.eval("x + y", %{"x" => 10, "y" => 5})
# => 15

# Work with strings
Pythonx.eval("name.upper()", %{"name" => "elixir"})
# => "ELIXIR"

Data Type Conversion

Pythonx automatically converts between Python and Elixir data types:

# Lists
Pythonx.eval("[1, 2, 3]")
# => [1, 2, 3]

# Dictionaries to maps
Pythonx.eval("{'name': 'Alice', 'age': 30}")
# => %{"name" => "Alice", "age" => 30}

# Nested structures
Pythonx.eval("{'users': [{'id': 1, 'name': 'Bob'}, {'id': 2, 'name': 'Carol'}]}")
# => %{"users" => [%{"id" => 1, "name" => "Bob"}, %{"id" => 2, "name" => "Carol"}]}

Working with Python Libraries

# NumPy operations
Pythonx.eval("""
import numpy as np
arr = np.array([1, 2, 3, 4, 5])
arr.mean()
""")
# => 3.0

# Pandas DataFrames
result = Pythonx.eval("""
import pandas as pd
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
df.sum().to_dict()
""")
# => %{"A" => 6, "B" => 15}

Practical Use Cases

1. Machine Learning Model Integration

One of the most compelling use cases is integrating existing Python ML models into Elixir applications:

defmodule MLPredictor do
  def load_model do
    Pythonx.eval("""
    import joblib
    model = joblib.load('model.pkl')
    """)
  end

  def predict(features) when is_list(features) do
    Pythonx.eval(
      """
      import numpy as np
      X = np.array(features).reshape(1, -1)
      prediction = model.predict(X)[0]
      confidence = model.predict_proba(X)[0].max()
      {'prediction': int(prediction), 'confidence': float(confidence)}
      """,
      %{"features" => features}
    )
  end
end

# Usage
MLPredictor.load_model()
MLPredictor.predict([5.1, 3.5, 1.4, 0.2])
# => %{"prediction" => 0, "confidence" => 0.95}

2. Data Science Workflows in Livebook

Pythonx shines in Livebook notebooks where you can mix Elixir’s data processing with Python’s visualization libraries:

# In a Livebook cell
data = MyApp.Repo.all(MyApp.Sale)
  |> Enum.map(fn sale ->
    %{date: sale.date, amount: sale.amount}
  end)

# Convert to Python and visualize
~PY"""
import matplotlib.pyplot as plt
import pandas as pd

df = pd.DataFrame($data)
df.plot(x='date', y='amount', kind='line')
plt.savefig('sales.png')
"""

3. Scientific Computing

Leverage Python’s scientific libraries for complex calculations:

defmodule ScientificCompute do
  def solve_linear_system(a_matrix, b_vector) do
    Pythonx.eval(
      """
      import numpy as np
      A = np.array(a_matrix)
      b = np.array(b_vector)
      x = np.linalg.solve(A, b)
      x.tolist()
      """,
      %{"a_matrix" => a_matrix, "b_vector" => b_vector}
    )
  end

  def fft_analysis(signal) do
    Pythonx.eval(
      """
      import numpy as np
      from scipy.fft import fft

      signal_array = np.array(signal)
      fft_result = fft(signal_array)
      {
        'magnitude': np.abs(fft_result).tolist(),
        'phase': np.angle(fft_result).tolist()
      }
      """,
      %{"signal" => signal}
    )
  end
end

4. Natural Language Processing

Use Python’s NLP libraries like spaCy or NLTK:

defmodule TextAnalyzer do
  def init do
    Pythonx.eval("""
    import spacy
    nlp = spacy.load('en_core_web_sm')
    """)
  end

  def extract_entities(text) do
    Pythonx.eval(
      """
      doc = nlp(text)
      entities = [
        {'text': ent.text, 'label': ent.label_}
        for ent in doc.ents
      ]
      entities
      """,
      %{"text" => text}
    )
  end

  def sentiment_analysis(text) do
    Pythonx.eval(
      """
      from textblob import TextBlob
      blob = TextBlob(text)
      {
        'polarity': blob.sentiment.polarity,
        'subjectivity': blob.sentiment.subjectivity
      }
      """,
      %{"text" => text}
    )
  end
end

# Usage
TextAnalyzer.init()
TextAnalyzer.extract_entities("Apple Inc. was founded in Cupertino by Steve Jobs.")
# => [
#   %{"text" => "Apple Inc.", "label" => "ORG"},
#   %{"text" => "Cupertino", "label" => "GPE"},
#   %{"text" => "Steve Jobs", "label" => "PERSON"}
# ]

5. Image Processing

Integrate Python’s powerful image processing libraries:

defmodule ImageProcessor do
  def resize_image(image_path, width, height) do
    Pythonx.eval(
      """
      from PIL import Image
      import io
      import base64

      img = Image.open(image_path)
      img_resized = img.resize((width, height))

      buffer = io.BytesIO()
      img_resized.save(buffer, format='PNG')
      buffer.seek(0)
      base64.b64encode(buffer.read()).decode()
      """,
      %{"image_path" => image_path, "width" => width, "height" => height}
    )
  end

  def detect_faces(image_path) do
    Pythonx.eval(
      """
      import cv2

      img = cv2.imread(image_path)
      gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

      face_cascade = cv2.CascadeClassifier(
        cv2.data.haarcascades + 'haarcascade_frontalface_default.xml'
      )

      faces = face_cascade.detectMultiScale(gray, 1.1, 4)
      [{'x': int(x), 'y': int(y), 'w': int(w), 'h': int(h)} for (x, y, w, h) in faces]
      """,
      %{"image_path" => image_path}
    )
  end
end

Important Limitations and Considerations

The Global Interpreter Lock (GIL)

The most critical limitation of Pythonx is Python’s Global Interpreter Lock (GIL). The GIL prevents multiple threads from executing Python code simultaneously, which has important implications:

Impact on Concurrency:

# This won't give you the concurrency you expect!
tasks = for i <- 1..10 do
  Task.async(fn ->
    Pythonx.eval("expensive_computation(#{i})")
  end)
end

Task.await_many(tasks)

Even though you’re spawning 10 Elixir processes, they’ll be serialized when executing Python code due to the GIL.

Solutions:

1. Use a Single Process: Route all Python calls through one GenServer

   defmodule PythonxServer do
     use GenServer
   
     def start_link(_) do
       GenServer.start_link(__MODULE__, nil, name: __MODULE__)
     end
   
     def eval(code, vars \\ %{}) do
       GenServer.call(__MODULE__, {:eval, code, vars})
     end
   
     def handle_call({:eval, code, vars}, _from, state) do
       result = Pythonx.eval(code, vars)
       {:reply, result, state}
     end
   end

2. Leverage Native Libraries: Packages like NumPy and Pandas release the GIL for their computationally intensive operations

   # These operations release the GIL and can run concurrently
   Pythonx.eval("numpy.dot(large_matrix_a, large_matrix_b)")

3. Use Ports for True Parallelism: For CPU-bound Python work that needs true parallelism:

   # Spawn multiple Python processes for parallel work
   tasks = for i <- 1..10 do
     Task.async(fn ->
       System.cmd("python", ["script.py", to_string(i)])
     end)
   end

Memory Considerations

Python objects live in the same process as your BEAM application:

• Large Python objects increase your BEAM process memory
• Python garbage collection is separate from BEAM’s
• Be mindful when working with large datasets

Error Handling

Python exceptions need proper handling:

defmodule SafePythonx do
  def safe_eval(code, vars \\ %{}) do
    try do
      {:ok, Pythonx.eval(code, vars)}
    rescue
      e -> {:error, Exception.message(e)}
    end
  end
end

case SafePythonx.safe_eval("1 / 0") do
  {:ok, result} -> result
  {:error, msg} -> Logger.error("Python error: #{msg}")
end

Best Practices

1. Minimize Python Call Frequency

# Bad: Multiple calls
def process_items(items) do
  Enum.map(items, fn item ->
    Pythonx.eval("process(item)", %{"item" => item})
  end)
end

# Good: Single batch call
def process_items(items) do
  Pythonx.eval(
    "[process(item) for item in items]",
    %{"items" => items}
  )
end

2. Initialize Heavy Resources Once

defmodule MLModel do
  use GenServer

  def start_link(_) do
    GenServer.start_link(__MODULE__, nil, name: __MODULE__)
  end

  def init(_) do
    # Load model once at startup
    Pythonx.eval("""
    import tensorflow as tf
    model = tf.keras.models.load_model('model.h5')
    """)
    {:ok, nil}
  end

  def predict(input) do
    GenServer.call(__MODULE__, {:predict, input})
  end

  def handle_call({:predict, input}, _from, state) do
    result = Pythonx.eval("model.predict(input)", %{"input" => input})
    {:reply, result, state}
  end
end

3. Use Type Specs for Safety

@spec analyze_sentiment(String.t()) :: {:ok, float()} | {:error, String.t()}
def analyze_sentiment(text) when is_binary(text) do
  try do
    score = Pythonx.eval(
      "TextBlob(text).sentiment.polarity",
      %{"text" => text}
    )
    {:ok, score}
  rescue
    e -> {:error, Exception.message(e)}
  end
end

When to Use Pythonx vs Alternatives

Use Pythonx When:

✅ Developing in Livebook and need Python libraries ✅ Prototyping ML integration before building production solutions ✅ Working with Python libraries that have no Elixir equivalent ✅ Calls are infrequent or can be batched ✅ Using Python libraries that release the GIL (NumPy, etc.) ✅ Simplicity matters more than maximum performance

Consider Alternatives When:

❌ You need true parallel execution of Python code ❌ Building high-throughput production systems ❌ Python operations are frequent and distributed across many processes ❌ You can use pure Elixir alternatives (Nx, Axon, Explorer) ❌ Your Python code is long-running or stateful

Alternative Approaches:

1. Nx/Axon for ML: Pure Elixir machine learning

# Native Elixir ML - no Python needed
model = Axon.input("input", shape: {nil, 784})
  |> Axon.dense(128, activation: :relu)
  |> Axon.dense(10, activation: :softmax)

2. Ports for Parallelism: True concurrent Python execution

Port.open({:spawn, "python ml_worker.py"}, [:binary])

3. HTTP APIs: Separate Python services

HTTPoison.post("http://python-service/predict", Jason.encode!(data))

4. Explorer for DataFrames: Elixir’s answer to Pandas

df = Explorer.DataFrame.new(%{a: [1, 2, 3], b: [4, 5, 6]})
Explorer.DataFrame.mutate(df, c: a + b)

Real-World Architecture Example

Here’s how you might architect a production system using Pythonx strategically:

defmodule MyApp.ML.Pipeline do
  # Use Elixir for orchestration, concurrency, and most processing
  def process_batch(records) do
    records
    |> Stream.chunk_every(100)
    |> Task.async_stream(&preprocess/1, max_concurrency: 10)
    |> Stream.map(fn {:ok, batch} -> batch end)
    # Single Python call for the ML inference step
    |> Enum.map(&predict_with_python/1)
    |> Stream.map(&postprocess/1)
    |> Enum.to_list()
  end

  # Elixir handles data transformation
  defp preprocess(batch) do
    Enum.map(batch, fn record ->
      %{
        features: extract_features(record),
        metadata: record.metadata
      }
    end)
  end

  # Python handles ML inference (called through single process)
  defp predict_with_python(batch) do
    PythonxServer.predict(batch)
  end

  # Elixir handles result processing
  defp postprocess(predictions) do
    Enum.map(predictions, &format_prediction/1)
  end
end

The Future of Python-Elixir Integration

The ecosystem continues to evolve:

• Pythonx is actively maintained by the Livebook team
• Explorer provides DataFrame functionality similar to Pandas
• Nx/Axon offer native ML capabilities
• Bumblebee brings pre-trained models from Hugging Face
• Community tools continue to improve interoperability

The trend is toward more native Elixir solutions while maintaining pragmatic Python integration where it makes sense.

Conclusion

Pythonx represents a pragmatic approach to Python-Elixir integration, particularly valuable for:

• Data science workflows in Livebook
• Rapid prototyping with Python libraries
• Scenarios where simplicity trumps maximum performance
• Leveraging Python’s extensive ML ecosystem

However, it’s not a silver bullet. The GIL limitations mean it’s not suitable for high-concurrency production systems that need to execute Python code in parallel. For those scenarios, consider ports, HTTP APIs, or increasingly mature native Elixir solutions.

The key is choosing the right tool for your specific use case. Pythonx opens doors to Python’s ecosystem while letting you stay within your Elixir application, making it an invaluable tool when used appropriately.

Getting Started

Ready to try Pythonx? Here’s your path forward:

1. Experiment in Livebook: Download Livebook and try the ~PY sigil
2. Read the docs: Check out Pythonx on HexDocs
3. Start small: Integrate one Python library into an existing Elixir project
4. Join the community: Discuss your use cases on Elixir Forum

Need Help Integrating Python with Elixir?

At AsyncSquad Labs, we specialize in building scalable Elixir applications and can help you navigate the Python-Elixir integration landscape. Whether you need to leverage existing Python ML models, migrate to native Elixir solutions, or architect a hybrid system, we can guide you to the right solution.

Contact us to discuss your integration needs and learn how we can help you combine the best of both ecosystems.

Related Articles

• Elixir and AI: Building Scalable Machine Learning Systems
• Building Real-Time Applications with Elixir and Phoenix: A Practical Guide
• Why Choose Elixir for Your Next Project?

📬 Stay Updated with Our Latest Insights

Get expert tips on software development, AI integration, and best practices delivered to your inbox. Join our community of developers and tech leaders.

Subscribe

We respect your privacy. Unsubscribe at any time. No spam, ever.