Model Copying and ORM Detachment¶

This guide explains LumiX’s strategy for safely copying optimization models that use ORM (Object-Relational Mapping) data sources, enabling what-if analysis and scenario comparison.

Overview¶

Why Model Copying is Needed¶

Many analysis workflows require creating modified copies of a model:

What-If Analysis: Test parameter changes without affecting the original model
Scenario Analysis: Compare multiple variations of the same base model
Sensitivity Analysis: Explore ranges of parameter values
A/B Testing: Compare different modeling approaches

The Challenge with ORM¶

When using database ORM frameworks (SQLAlchemy, Django), model copying faces two challenges:

Session Binding: ORM objects are bound to database sessions and cannot be pickled
Lazy Loading: Related data may not be loaded yet, causing errors after copying

Example Problem:

from copy import deepcopy

# Build model with SQLAlchemy data
product = session.query(Product).first()
production = LXVariable[Product, float]("production")
    .indexed_by(lambda p: p.id)
    .from_model(session)

model.add_variable(production)

# This will FAIL with ORM session errors!
modified_model = deepcopy(model)  # ❌ Error: Cannot pickle session

LumiX’s Solution¶

ORM Detachment Strategy¶

LumiX implements automatic ORM detachment in __deepcopy__ methods:

        graph TD
    A[deepcopy model] --> B{Contains ORM objects?}
    B -->|Yes| C[Materialize lazy data]
    C --> D[Detach from session]
    D --> E[Copy as plain objects]
    B -->|No| E
    E --> F[Safe to copy!]

    style A fill:#e8f4f8
    style F fill:#e1ffe1

Three-Step Process:

Detect ORM Objects: Identify SQLAlchemy or Django ORM instances
Materialize Data: Force-load lazy relationships before copying
Detach from Session: Create plain Python objects with same attributes

How It Works¶

The strategy is transparent - just use deepcopy normally:

from copy import deepcopy
from lumix import LXModel, LXVariable

# Build model with ORM data (session-bound objects)
model = LXModel("production")
production = LXVariable[Product, float]("production")
    .from_model(session)  # Uses SQLAlchemy session

model.add_variable(production)

# This now works! ORM objects automatically detached
modified_model = deepcopy(model)  # ✓ Success

# Modified model is independent (no session)
modified_model.constraints[0].rhs_value = 1500  # Safe to modify

Supported ORM Frameworks¶

LumiX automatically detects and handles:

Framework	Detection	Detachment Strategy
SQLAlchemy	`hasattr(obj, '_sa_instance_state')`	Create new instance, copy column attributes
Django ORM	`hasattr(obj, '_state') and hasattr(obj, '_meta')`	Copy field values to new instance
Plain Python	N/A	Return as-is (no detachment needed)

Implementation Details¶

Core Utility Functions¶

The lumix.utils.copy_utils module provides:

detach_orm_object¶

Detach a single ORM object from its database session.

from lumix.utils.copy_utils import detach_orm_object

# With SQLAlchemy
product = session.query(Product).first()
detached = detach_orm_object(product)
# detached is now a plain Python object, safe to pickle

# With plain objects (no-op)
plain_obj = PlainProduct(id=1, name="Chair")
result = detach_orm_object(plain_obj)
# result is plain_obj (same object, unchanged)

How it works for SQLAlchemy:

Create new instance of same class (cls.__new__(cls))
Initialize __dict__ to make it a plain Python object
Copy all column attribute values as plain attributes
Copy loaded relationship attributes (if already loaded)
Return plain object with no session binding

Signature:

def detach_orm_object(obj: Any) -> Any:
    """
    Detach ORM object from session, making it safe to copy.

    Args:
        obj: Object to detach (ORM or plain object)

    Returns:
        Detached copy (ORM) or original object (plain Python)
    """

materialize_and_detach_list¶

Materialize and detach a list of items that may contain ORM objects.

from lumix.utils.copy_utils import materialize_and_detach_list

# List of SQLAlchemy objects
products = session.query(Product).all()
detached_list = materialize_and_detach_list(products, {})
# Each item is now detached and deep copied

Signature:

def materialize_and_detach_list(
    items: Optional[List[Any]],
    memo: dict
) -> Optional[List[Any]]:
    """
    Materialize and detach list of items.

    Args:
        items: List of items (may contain ORM objects), or None
        memo: deepcopy memo dict for circular reference tracking

    Returns:
        New list with detached and deep-copied objects, or None
    """

copy_function_detaching_closure¶

Copy a function while detaching any ORM objects in its closure.

This is critical for lambda functions that capture ORM objects:

from lumix.utils.copy_utils import copy_function_detaching_closure

# Lambda capturing ORM object
product = session.query(Product).first()  # ORM object
profit_func = lambda p: product.profit_per_unit * p.quantity

# Create safe copy
safe_func = copy_function_detaching_closure(profit_func, {})
# safe_func uses detached copy of 'product'

How it works:

Check if function has a closure
Inspect each cell in the closure
Detect ORM objects in closure variables
Detach ORM objects
Create new function with detached closure

Signature:

def copy_function_detaching_closure(
    func: Callable,
    memo: dict
) -> Callable:
    """
    Copy function while detaching ORM objects in closure.

    Args:
        func: Function to copy (may have ORM objects in closure)
        memo: deepcopy memo dict for circular reference tracking

    Returns:
        New function with ORM objects detached from sessions
    """

Integration in Core Classes¶

LumiX integrates ORM detachment into __deepcopy__ methods of core classes:

LXModel.deepcopy¶

from lumix.core.model import LXModel

class LXModel:
    def __deepcopy__(self, memo):
        """Custom deepcopy that detaches ORM sessions."""
        # ... create new instance ...

        # Deep copy all variables (calls LXVariable.__deepcopy__)
        result.variables = [deepcopy(var, memo) for var in self.variables]

        # Deep copy all constraints (calls LXConstraint.__deepcopy__)
        result.constraints = [deepcopy(c, memo) for c in self.constraints]

        # Deep copy objective expression
        result.objective_expr = deepcopy(self.objective_expr, memo)

        return result

LXVariable.deepcopy¶

from lumix.core.variables import LXVariable

class LXVariable:
    def __deepcopy__(self, memo):
        """Custom deepcopy that detaches ORM and handles closures."""
        from ..utils.copy_utils import (
            materialize_and_detach_list,
            copy_function_detaching_closure
        )

        # ... create new instance ...

        # Copy callable attributes (may have closures with ORM objects)
        result.index_func = copy_function_detaching_closure(
            self.index_func, memo
        ) if self.index_func is not None else None

        result.cost_func = copy_function_detaching_closure(
            self.cost_func, memo
        ) if self.cost_func is not None else None

        # Handle data sources
        if self._session is not None:
            # Materialize ORM data before copying
            instances = self.get_instances()
            result._data = materialize_and_detach_list(instances, memo)
            result._session = None  # Clear session reference
        elif self._data is not None:
            # Already have data - just detach and copy
            result._data = materialize_and_detach_list(self._data, memo)
            result._session = None

        return result

LXConstraint.deepcopy¶

Similar strategy for constraints:

from lumix.core.constraints import LXConstraint

class LXConstraint:
    def __deepcopy__(self, memo):
        """Custom deepcopy that detaches ORM in expressions."""
        # ... create new instance ...

        # Deep copy expression (handles ORM in coefficients)
        result.expr = deepcopy(self.expr, memo) if self.expr else None

        return result

Usage Examples¶

Basic Usage¶

Simple model copying:

from copy import deepcopy
from lumix import LXModel, LXVariable, LXOptimizer

# Build model with ORM data
session = get_session()
model = LXModel("production")

production = LXVariable[Product, float]("production")
    .continuous()
    .bounds(lower=0)
    .indexed_by(lambda p: p.id)
    .from_model(session)

model.add_variable(production)

# Copy model (ORM automatically detached)
modified_model = deepcopy(model)

# Safe to modify
modified_model.constraints[0].rhs_value *= 1.5

# Solve both
optimizer = LXOptimizer()
original_solution = optimizer.solve(model)
modified_solution = optimizer.solve(modified_model)

What-If Analysis¶

Using model copying for what-if analysis:

from copy import deepcopy
from lumix import LXWhatIfAnalyzer

# LXWhatIfAnalyzer uses deepcopy internally
analyzer = LXWhatIfAnalyzer(model, optimizer)

# Each what-if creates a modified copy
result = analyzer.increase_constraint_rhs("capacity", by=100)

# Behind the scenes:
# 1. deepcopy(model) - uses ORM detachment
# 2. Modify copied model
# 3. Solve modified model
# 4. Compare results

Scenario Analysis¶

Multiple model copies for scenarios:

from copy import deepcopy

scenarios = {}

# Optimistic scenario
optimistic = deepcopy(model)
optimistic.get_constraint("demand").rhs_value *= 1.2
scenarios["optimistic"] = optimizer.solve(optimistic)

# Baseline scenario
scenarios["baseline"] = optimizer.solve(model)

# Pessimistic scenario
pessimistic = deepcopy(model)
pessimistic.get_constraint("demand").rhs_value *= 0.8
scenarios["pessimistic"] = optimizer.solve(pessimistic)

# Compare
for name, solution in scenarios.items():
    print(f"{name}: ${solution.objective_value:,.2f}")

Manual ORM Detachment¶

If you need to manually detach objects:

from lumix.utils.copy_utils import detach_orm_object

# Detach single object
product = session.query(Product).first()
detached_product = detach_orm_object(product)

# Now safe to use outside session
session.close()
print(detached_product.name)  # ✓ Works

Lambda with ORM in Closure¶

Handling lambdas that capture ORM objects:

from copy import deepcopy

# Lambda captures ORM object
product = session.query(Product).first()  # Session-bound

production = LXVariable[Product, float]("production")
    .continuous()
    .indexed_by(lambda p: p.id)
    .from_data([product])

# Add coefficient function that captures 'product'
expr = LXLinearExpression()
expr.add_term(production, lambda p: product.profit_per_unit)  # Captures 'product'

# Deep copy handles this automatically!
expr_copy = deepcopy(expr)  # ✓ Works - 'product' detached in closure

Advanced Topics¶

Performance Considerations¶

Materialization Cost: Lazy-loaded relationships are materialized during detachment, which can be expensive for large datasets.

Optimization Strategies:

Eager Loading: Use .options(joinedload(...)) in queries
Limit Data: Only query needed columns
Cache Results: Reuse detached objects when possible

from sqlalchemy.orm import joinedload

# Eager load relationships
products = session.query(Product).options(
    joinedload(Product.materials),
    joinedload(Product.machine_requirements)
).all()

# Now all data is loaded, detachment is faster
production = LXVariable[Product, float]("production")
    .from_data(products)

model_copy = deepcopy(model)  # Faster with eager loading

Circular References¶

The memo dict in deepcopy handles circular references:

# Circular reference example
class Node:
    def __init__(self, value):
        self.value = value
        self.next = None

# Create cycle
node1 = Node(1)
node2 = Node(2)
node1.next = node2
node2.next = node1  # Circular!

# deepcopy handles this with memo dict
node1_copy = deepcopy(node1)  # ✓ Works

LumiX uses the same memo dict throughout the copy process.

Pickle Support¶

In addition to __deepcopy__, LumiX implements __getstate__ and __setstate__ for pickle protocol:

import pickle

# Build model with ORM data
model = build_production_model(session)

# Pickle (uses __getstate__ for ORM detachment)
pickled = pickle.dumps(model)

# Unpickle (uses __setstate__ for restoration)
restored_model = pickle.loads(pickled)

# Model works without session
solution = optimizer.solve(restored_model)

Troubleshooting¶

Session Errors After Copying¶

Symptom: DetachedInstanceError or Session is closed errors after copying

Cause: Object not properly detached from session

Solution: Ensure you’re using deepcopy, not copy

from copy import copy, deepcopy

# Bad - shallow copy doesn't detach
bad_model = copy(model)  # ❌

# Good - deep copy detaches
good_model = deepcopy(model)  # ✓

Missing Data After Copying¶

Symptom: Copied model has None or empty lists for ORM data

Cause: Lazy-loaded relationships not materialized before copying

Solution: Eager load relationships or access them before copying

from sqlalchemy.orm import joinedload

# Option 1: Eager loading
products = session.query(Product).options(
    joinedload(Product.materials)
).all()

# Option 2: Touch lazy attributes before copying
for product in products:
    _ = product.materials  # Force load
    _ = product.machine_requirements  # Force load

# Now copy will include all data
model_copy = deepcopy(model)

Lambda Closure Issues¶

Symptom: PicklingError mentioning lambda or closure

Cause: Lambda closure contains un-picklable objects

Solution: Use copy_function_detaching_closure or avoid capturing complex objects

from lumix.utils.copy_utils import copy_function_detaching_closure

# Problem: Lambda captures unpicklable object
session_obj = session  # Session cannot be pickled
bad_func = lambda p: session_obj.query(...)  # ❌

# Solution: Don't capture session in lambda
good_func = lambda p: p.profit_per_unit  # ✓

Best Practices¶

Use Eager Loading

Load all needed data before copying to avoid lazy-loading errors.

Close Sessions Before Copying

Detachment makes session unnecessary - close it for clarity.

# Build model
model = build_model(session)

# Close session (model now uses detached data)
session.close()

# Safe to copy
model_copy = deepcopy(model)

Avoid Complex Closures

Keep lambda functions simple to avoid pickling issues.

# Bad: Complex closure
def make_cost_func(session, product_id):
    product = session.query(Product).get(product_id)
    return lambda p: product.cost * p.quantity  # ❌

# Good: Simple lambda
def make_cost_func(product):
    cost = product.cost  # Capture value, not object
    return lambda p: cost * p.quantity  # ✓

Test Copying Early

Verify copying works before building complex models.

# Build minimal model
model = LXModel("test")
# ... add variables ...

# Test copying
try:
    model_copy = deepcopy(model)
    print("✓ Copying works!")
except Exception as e:
    print(f"❌ Copy failed: {e}")

Use Type Hints

Help IDE and type checkers understand ORM types.

from typing import List
from sqlalchemy.orm import Session

def build_model(session: Session) -> LXModel:
    products: List[Product] = session.query(Product).all()
    # Type checker knows products is List[Product]

References¶

SQLAlchemy Session: https://docs.sqlalchemy.org/en/latest/orm/session.html
Django ORM: https://docs.djangoproject.com/en/stable/topics/db/queries/
Python deepcopy: https://docs.python.org/3/library/copy.html

Model Copying and ORM Detachment¶

Overview¶

Why Model Copying is Needed¶

The Challenge with ORM¶

LumiX’s Solution¶

ORM Detachment Strategy¶

How It Works¶

Supported ORM Frameworks¶

Implementation Details¶

Core Utility Functions¶

detach_orm_object¶

materialize_and_detach_list¶

copy_function_detaching_closure¶

Integration in Core Classes¶

LXModel.deepcopy¶

LXVariable.deepcopy¶

LXConstraint.deepcopy¶

Usage Examples¶

Basic Usage¶

What-If Analysis¶

Scenario Analysis¶

Manual ORM Detachment¶

Lambda with ORM in Closure¶

Advanced Topics¶

Performance Considerations¶

Circular References¶

Pickle Support¶

Troubleshooting¶

Session Errors After Copying¶

Missing Data After Copying¶

Lambda Closure Issues¶

Best Practices¶

See Also¶

API Reference¶

References¶

Model Copying and ORM Detachment¶

Overview¶

Why Model Copying is Needed¶

The Challenge with ORM¶

LumiX’s Solution¶

ORM Detachment Strategy¶

How It Works¶

Supported ORM Frameworks¶

Implementation Details¶

Core Utility Functions¶

detach_orm_object¶

materialize_and_detach_list¶

copy_function_detaching_closure¶

Integration in Core Classes¶

LXModel.__deepcopy__¶

LXVariable.__deepcopy__¶

LXConstraint.__deepcopy__¶

Usage Examples¶

Basic Usage¶

What-If Analysis¶

Scenario Analysis¶

Manual ORM Detachment¶

Lambda with ORM in Closure¶

Advanced Topics¶

Performance Considerations¶

Circular References¶

Pickle Support¶

Troubleshooting¶

Session Errors After Copying¶

Missing Data After Copying¶

Lambda Closure Issues¶

Best Practices¶

See Also¶

API Reference¶

References¶

LXModel.deepcopy¶

LXVariable.deepcopy¶

LXConstraint.deepcopy¶