EcoPrune is a framework for automatic hidden layer size discovery in neural networks, inspired by biological ecosystems:
- Apoptosis: True structural pruning of diseased connections
- Composting: Recycling acquired knowledge
- Neurogenesis: New growth from recycled compost materials
Core Hypothesis: Starting training with an over-parameterized network then gradually pruning it with fine-tuning discovers hidden sizes that outperform manual design.
| Model | Parameters | Accuracy | Notes |
|---|---|---|---|
| Large Baseline | 42,115 | 71.5% | Manual design |
| EcoPrune (Ours) | 23,372 | 72.8% | Auto-discovery |
| Same-Size Baseline | 23,372 | 70.0% | Manual design (same size) |
Summary: The ecological model outperforms manual same-size design by +2.8%, and the large model by +1.3% while reducing parameters by 44.5%.
Instead of measuring weight magnitude alone, we use sensitivity:
health_ij = EMA(|weight_ij * gradient_ij|)
This measures how much a weight affects the loss, not its absolute size.
We delete entire rows (output neurons) with low health, and actually reshape the matrices — not just zero them out.
Deleted weights are not wasted. They are saved in a Compost Heap and their statistical distribution is reused to initialize new connections.
from ecoprune import EcoNet, train_ecological, make_spiral
import torch
# 1. Create ecological network
model = EcoNet([2, 256, 128, 64, 3])
# 2. Prepare data
X_train, y_train = make_spiral(400, 3, 0.25)
X_test, y_test = make_spiral(200, 3, 0.25)
# 3. Ecological training
history = train_ecological(
model, X_train, y_train, X_test, y_test,
total_epochs=200,
prune_epochs=[60, 100, 140],
prune_quantile=0.10,
finetune_epochs=30,
lr=0.02
)
# 4. Results
print(f"Final structure: {model.current_sizes}")
print(f"Parameters: {model.count_params():,}")Epoch 0-60 : Initial training (knowledge acquisition)
Epoch 60 : Light pruning (10%) + 30 epochs fine-tuning
Epoch 60-100 : Stabilization
Epoch 100 : Second pruning + fine-tuning
Epoch 100-140: Stabilization
Epoch 140 : Third pruning + fine-tuning
Epoch 140-200: Final stabilization
| File | Description |
|---|---|
ecoprune.py |
Complete source code |
ecoprune_v2_final.png |
Results visualization |
final_ecological_summary.png |
Visual summary |
ecological_network_evolution.png |
Network evolution over time |
README.md |
This file |
⚠️ This is a Research Prototype, not a production-ready tool.
✅ Structural pruning works (actual matrix shrinkage)
✅ Sensitivity-based health tracking outperforms fixed thresholds
✅ Auto-discovery outperforms manual same-size design
🔧 Testing on MNIST / CIFAR-10
🔧 Comparison with Lottery Ticket Hypothesis
🔧 Fine-tuning schedule optimization
🔧 Support for Convolutional Neural Networks (CNNs)
- Frankle, J., & Carlin, N. (2018). The Lottery Ticket Hypothesis. ICML.
- Han, S., Mao, H., & Dally, W. (2015). Deep Compression. ICLR.
- Molchanov, P., et al. (2019). Importance Estimation for Neural Network Pruning. CVPR.
MIT License - Open source for everyone.
Made with ❤️ and Science 🔬