🏛️ Westminster College, London | wcc.co.uk

📊 Big Data & Quantum Analytics · Advanced Notes

🗓️ 6 Saturdays | 9am–11am 🎯 Intermediate → Advanced 💻 Online · Live + Hands‑on

Converging Big Data engineering (Apache Spark, HDFS) with quantum computing, QUBO models, hybrid quantum ML, quantum annealing, tensor networks, and production pipelines on AWS Braket.

📦 Module 1 — Big Data Engineering & Quantum Computing Foundations Week 1 · 2h

🏗️ Big Data Ecosystems & Apache Spark

Big data processing requires distributed frameworks. Apache Spark provides in‑memory cluster computing with APIs in Python (PySpark), Scala, and SQL. HDFS (Hadoop Distributed File System) stores data across nodes with replication for fault tolerance. Spark reads from HDFS, S3, or local. Key concepts: RDDs (Resilient Distributed Datasets), DataFrames, Spark SQL, lazy evaluation, and transformations (map, filter, reduceByKey).

# PySpark: read CSV and compute aggregate

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("BigDataQuantum").getOrCreate()

df = spark.read.csv("hdfs://data/sales.csv", header=True)

df.groupBy("region").sum("revenue").show()

⚛️ Quantum Foundations: Qubits, Gates, and Circuits

Quantum computing leverages superposition (|ψ⟩=α|0⟩+β|1⟩), entanglement (non‑local correlations), and interference. Single‑qubit gates: X (NOT), H (Hadamard), Z, S, T. Multi‑qubit gates: CNOT, CZ, SWAP. Quantum circuits are sequences of gates applied to qubits. Hardware platforms: superconducting (IBM, Google), trapped ions (IonQ), photonics. For big data analytics, quantum algorithms can offer quadratic or exponential speedups for specific problems.

# Qiskit example: superposition + entanglement

from qiskit import QuantumCircuit

qc = QuantumCircuit(2)

qc.h(0); qc.cx(0,1)

qc.draw()

🔁 Building Scalable Data Pipelines for Quantum‑Enhanced Analytics

Data pipelines extract, transform, and load (ETL) large datasets into formats suitable for quantum or hybrid processing. Example: use Spark to preprocess and reduce dimensionality, then convert to QUBO or feature maps for quantum circuits. Store intermediates in Parquet/ORC. Connect to quantum backends via cloud APIs (IBMQ, AWS Braket).

🎯 Module 2 — QUBO Models & Large‑Scale Optimisation Week 2 · 2h

📐 Quadratic Unconstrained Binary Optimisation (QUBO)

QUBO formulation: minimize xᵀ Q x where x ∈ {0,1}ⁿ and Q is an n×n real matrix. Many NP‑hard problems (MaxCut, vertex cover, portfolio optimization) map naturally to QUBO. Reformulate constraints as penalty terms: constraint + λ·(violation)². QUBO is equivalent to Ising model: sᵢ = 2xᵢ−1 ∈ {−1,+1}. Quantum annealers (D‑Wave) and QAOA directly solve QUBO problems.

# Example QUBO for MaxCut on 3 vertices

# Maximize sum_{(i,j) in E} (1 - s_i s_j)/2 → minimize s^T W s

# Convert to QUBO: Q matrix

import numpy as np

Q = np.array([[-0.5, 1, 1], [0, -0.5, 1], [0, 0, -0.5]])  # upper triangular

# Solve via simulated annealing or quantum annealer

🔥 Simulated Annealing & Quantum‑Inspired Optimisation

Simulated annealing (SA) is a classical heuristic for combinatorial optimisation, inspired by metallurgy. Quantum annealing (QA) uses quantum tunnelling to escape local minima. Quantum‑inspired algorithms run on classical hardware but mimic quantum effects (e.g., path integral Monte Carlo, tensor network optimisation). Applications: logistics (vehicle routing), scheduling (job shop), finance (portfolio optimisation), and supply chain.

💡 Benchmarking: Compare SA, QA, and classical solvers (Gurobi, OR‑Tools) on large‑scale instances to measure scalability and solution quality.

🧠 Module 3 — Hybrid Quantum Machine Learning Week 3 · 2h

⚙️ Hybrid Architectures & PennyLane

Hybrid quantum-classical ML uses parameterized quantum circuits (variational circuits) as layers within classical neural networks. PennyLane is a framework for differentiable quantum programming, integrating with PyTorch, TensorFlow, and JAX. Quantum nodes compute expectation values; gradients are computed via parameter shift rule. Variational quantum eigensolver (VQE) and quantum neural networks (QNNs) are core models.

import pennylane as qml

import torch

dev = qml.device('default.qubit', wires=2)

@qml.qnode(dev, interface='torch')

def circuit(x, weights):

    qml.AngleEmbedding(x, wires=[0,1])

    qml.BasicEntanglerLayers(weights, wires=[0,1])

    return qml.expval(qml.PauliZ(0))

# Hybrid model

class HybridModel(torch.nn.Module):

    def __init__(self):

        super().__init__()

        self.clayer = torch.nn.Linear(2, 2)

        self.qweights = torch.randn(2,2, requires_grad=True)

    def forward(self, x):

        x = self.clayer(x)

        return circuit(x, self.qweights)

🧬 Quantum Kernels & Feature Maps

Quantum kernels map classical data to quantum Hilbert space, enabling potentially richer similarity measures. A quantum feature map φ(x) = U(x)|0⟩. Kernel K(x,x') = |⟨φ(x)|φ(x')⟩|². Used in quantum support vector machines (QSVM). Can provide advantage over classical kernels for certain datasets (provable separation).

# Quantum kernel with PennyLane

dev = qml.device('default.qubit', wires=2)

@qml.qnode(dev)

def kernel_circuit(x1, x2):

    qml.AngleEmbedding(x1, wires=[0,1])

    qml.adjoint(qml.AngleEmbedding)(x2, wires=[0,1])

    return qml.probs(wires=[0,1])

# Kernel = probability of |00⟩ after interference

🏭 Module 4 — Quantum Annealing & Industrial Optimisation Week 4 · 2h

🔧 D‑Wave Systems & Quantum Annealing Workflows

Quantum annealers solve Ising/QUBO problems by adiabatic evolution. D‑Wave Advantage has >5000 qubits with Pegasus topology. Workflow: formulate problem as QUBO → embed onto hardware graph → anneal → read solution. Hybrid solvers combine classical pre/post‑processing with quantum sampling, scaling to millions of variables. Applications: supply chain, vehicle routing, job scheduling, energy grid optimization.

# D‑Wave Ocean SDK example

from dwave.system import DWaveSampler, EmbeddingComposite

sampler = EmbeddingComposite(DWaveSampler())

Q = {('x1','x1'): -1, ('x2','x2'): -1, ('x1','x2'): 2}

sampleset = sampler.sample_qubo(Q, num_reads=100)

print(sampleset.first)

📦 Real‑World Case Studies

Supply chain optimisation: minimise transportation cost + warehouse selection (QUBO).
Vehicle routing: assign deliveries to fleet with time windows → quadratic assignment.
Manufacturing scheduling: job shop with sequence‑dependent setup times.
Energy optimisation: microgrid scheduling, EV charging coordination.

📈 Performance: Hybrid solvers often outperform classical heuristics for large, constrained problems; advantage grows with problem size.

📐 Module 5 — Tensor Networks & Quantum Data Representation Week 5 · 2h

🧩 Tensor Algebra & Matrix Product States (MPS)

Tensors generalise matrices to higher dimensions. Tensor networks (TN) represent high‑dimensional data with low‑rank factorization. Matrix Product States (MPS) efficiently represent 1D quantum systems and large‑scale data tensors. TN compression reduces memory from exponential to polynomial, enabling analysis of massive datasets. Applications: quantum many‑body physics, machine learning (TN classifiers), image/video compression.

# Using TensorLy and quimb

import tensornetwork as tn

# Create a random MPS for 10 sites, bond dimension 4

nodes = [tn.Node(np.random.rand(2,4,4)) for _ in range(10)]

# Contract to compute expectation value

result = tn.contractors.auto(nodes).tensor

📉 Dimensionality Reduction & Data Encoding

Tensor networks can perform PCA‑like compression: decompose a large matrix into a product of smaller cores (tensor train decomposition). Quantum data encoding: amplitude encoding (2ⁿ amplitudes), angle encoding, or Hamiltonian encoding. Tensor networks also simulate quantum circuits classically (e.g., MPS for 1D circuits), enabling hybrid workflows.

☁️ Module 6 — Production Quantum Pipelines & Capstone Project Week 6 · 2h

🚀 AWS Braket & Hybrid Deployment

AWS Braket provides managed quantum computing services: simulators, on‑demand quantum hardware (IonQ, Rigetti, Oxford Quantum Circuits, and D‑Wave). Workflow: define circuit (using Braket SDK), choose device, submit task, retrieve results. Hybrid jobs: run classical pre‑processing on EC2, quantum task on Braket, then post‑process. Integrate with SageMaker for ML pipelines. Scalable orchestration: AWS Step Functions, Lambda, or Airflow for production pipelines.

# AWS Braket hybrid job (simulator)

from braket.circuits import Circuit

from braket.devices import LocalSimulator

from braket.aws import AwsDevice

circ = Circuit().h(0).cnot(0,1)

device = LocalSimulator()

result = device.run(circ, shots=1000).result()

# For real hardware: device = AwsDevice("arn:aws:braket:...")

📊 Monitoring, Security & Governance

Production quantum pipelines require tracking: job duration, qubit usage, cost (per task for cloud quantum). Security: quantum key distribution (QKD) for encrypted data, but hybrid pipelines follow standard IAM roles and encryption at rest. Governance: audit logs, versioning of quantum circuits (e.g., using Git + Braket's circuit versioning).

🎓 Capstone Project — Deploy a Quantum‑Inspired Model on AWS Braket

Project options:

Portfolio optimisation using QUBO → solve via D‑Wave hybrid solver on Braket.
Hybrid quantum neural network for classification (PennyLane + PyTorch) and deploy inference as Lambda function.
Tensor network compression of large image dataset (e.g., MNIST) with MPS classifier.
Supply chain routing solved with quantum annealing, with data pipeline using Spark pre‑processing and results stored in S3.

Deliverables: GitHub repo with infrastructure as code (Terraform/CDK), Jupyter notebooks, API endpoint (optional), and final presentation.

🏆 Assessment criteria: problem complexity, hybrid integration, scalability, performance comparison (classical vs quantum), and clear documentation.