> For the complete documentation index, see [llms.txt](https://youness-organization-4.gitbook.io/recommendation-system-infrastructure/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://youness-organization-4.gitbook.io/recommendation-system-infrastructure/project-architecture.md).

# Project Architecture

### Project Overview

We are developing a comprehensive MLOps project focused on crafting a recommendation system for rental clients. This system will efficiently suggest potential clients to stakeholders using advanced machine learning models.

### Technology Stack

* **Data Management:** Nifi, Feature Store
* **Machine Learning Frameworks:** TensorFlow, PyTorch
* **Version Control & CI/CD:** Git, Azure DevOps
* **Containerization:** Docker
* **Model Management:** MLflow
* **Storage Solutions:** Minio, Azure Blob Storage
* **Web Framework:** FastAPI
* **Programming Language:** Python
* **Monitoring & Logging:** Grafana, ELK Stack

### Objectives

* To build a recommendation system delivering high-quality client suggestions.
* To implement a scalable and robust MLOps pipeline.
* To utilize the best practices of continuous integration and deployment.

The architecture used a design pattern called **Orchestrated Pull-Based Training Architecture** which is designed for scenarios requiring scheduled model retraining. With data in a warehouse, a workflow orchestration tool is employed to schedule data extraction, processing, and retraining using fresh data. This setup is ideal for problems where real-time scoring isn't necessary, such as a content recommendation engine for songs or articles that delivers pre-computed recommendations.

<figure><img src="/files/PoiHD3jbrrr1CDutdDRR" alt=""><figcaption></figcaption></figure>

### Expected Outcomes

* Enhanced decision-making capabilities for stakeholders.
* Streamlined data and workflow management.
* Efficient and scalable deployment of machine learning models.

**Project Architecture**&#x20;

That's an overview of the project :&#x20;

<figure><img src="https://lh7-rt.googleusercontent.com/slidesz/AGV_vUdAHVsVLSNiXh3s6kjmf11WlUC-Rw1TcocYyUvtkTuFaHh2Hc0hrnS7logZl9cuCYyjakkZOhg2MgYeUhkcNCYkaw9RUuJA1rlCva6ffCLPELkQormjVUsOv93Gl_lu9NWlveWkSC5pmdmF2qOQqe7SWSgv6bg8=s2048?key=IKYIVPf_y5hnX4HNmjmMjg" alt=""><figcaption></figcaption></figure>

The project architecture consists of several key components:

1. **Data Ingestion Layer**: Responsible for collecting and processing raw data from various sources. This layer ensures data is cleaned and transformed for further use.
2. **Data Storage**: Utilizes scalable databases for storing both raw and processed data. Ensures data is easily accessible and secure.
3. **Data Processing Framework**: Employs robust data processing tools to analyze and prepare data for machine learning models.
4. **Machine Learning Models**: Includes training, testing, and deploying ML models. Models are constantly updated with new data for improved accuracy.
5. **Application Interface**: Provides users with interactive dashboards and tools to interact with data and insights generated by the ML models.
6. **Security Layer**: Ensures that all data and interactions with the architecture are secure and comply with relevant regulations and standards.

The solution will be hosted on a Virtual Machine (VM) in Azure servers for staging and pre-production testing. This will facilitate collaboration with the development team and involves a setup phase that includes installing the software stack, dependencies, secure networks, etc