SENIOR IT DATA SPECIALIST

Herman Teng

Data Engineering · BI · AI-Enabled Analytics

View My Work ↓ Get in Touch
0+
Years Exp
0+
Key Projects
0+
Tech Posts

Driving Data Transformation

Senior data professional with 10+ years designing enterprise-grade data platforms, BI solutions, and Lakehouse frameworks in Tier-1 banking. Proven track record unifying fragmented data landscapes, boosting operational efficiency, and governing high-stakes financial data.

Currently pioneering AI-augmented engineering and modernizing data architectures using Databricks, Unity Catalog, and Azure ecosystems to deliver scalable, secure analytics solutions.

0%
Infrastructure Monitoring

Continuously monitored critical infrastructure performance including ETL processing and capacity planning, maintaining 90% SLA compliance.

0x
Team Productivity

Designed automation for large-scale data access requests, reducing daily effort from 2.5h to <30m.

0%
Efficiency Gain

Led community initiatives enabling non-technical users to leverage advanced analytics across business teams.

Skills & Technologies

AI / GenAI

GitHub Copilot Copilot Studio MCP Servers Agent Skills GPT Claude Gemini RAG

Data Engineering & Lakehouse

Databricks Delta Lake PySpark Spark SQL Unity Catalog DAF2.0 ETL/ELT

BI & Visualization

Power BI Premium DAX Power Query Tableau SSRS Cognos

Cloud & Platforms

Azure ADLS Synapse SQL DB Fabrics Purview SQL Server Hadoop

Programming

Python T-SQL PySpark SAS Shell PowerShell VBA JS/TS

Data Governance

DAF Unity Catalog PIA/DIA EDC Purview RBAC Data Quality

Professional Experience

Senior IT Data Specialist (Promoted)
TD Bank · Enterprise Information Management · May 2022–Present
Click to view achievements
  • Platform Modernization: Led technical migration from legacy systems (Hadoop/SAS/Tableau) to Microsoft Azure Cloud (ADLS/Synapse/Databricks/Power BI Premium).
  • BISOPS Triage Solution: Engineered Python-based operational triage tracking dashboard (Pandas/Power BI/PySpark), saving 2 hours/day and reducing MTTR by 20%.
  • RMR2.0 Lakehouse Framework: Spearedheaded POC for Regulatory Margin Reporting (RMR) using Databricks Delta Lake and Unity Catalog. Validated end-to-end processing efficiency.
  • SRZ Synapse Inventory: Designed robust data onboarding pipeline into Azure Synapse SQL Pool, increasing data availability rate to 99.5%.
  • Enterprise Data Catalog: Implemented Microsoft Purview for automated lineage mapping and PII/PCI classification.
  • Data Asset Framework (DAF2.0): Designed Data Impact Assessment (DIA) methodology aligned with CDAO policies.
  • AI-Augmented Engineering: Pioneered use of GenAI (GitHub Copilot, custom LLM scripts) to accelerate code conversion, documentation, and reverse-engineering of legacy SAS/SQL.
Business Data Management Specialist
TD Bank · Treasury & Balance Sheet Management (TBSM) · May 2021–May 2022
Click to view achievements
  • Project Granite: Led end-to-end SIT, parallel testing, and production validation for massive retail product migration to target Liquidity & Interest Rate Risk platforms.
  • POD Volume Report: Redesigned critical balance mismatch reporting pipeline, replacing manual Excel extracts with automated SQL Server SSIS/SSRS workflows.
  • OSFI & Liquidity Reporting: Governed source-to-target mappings (STM) and ensured data quality for regulatory reporting engines (Condor/Indigo).
Senior BIM Analyst
TD Bank · Credit Cards · Aug 2017–May 2021
Click to view achievements
  • Merchant Solutions Migration: Drove data migration of $30B+ portfolio from legacy First Data to TSYS platform. Orchestrated validation scripts resulting in zero-defect cutover.
  • Runway BCC Implementation: Architected Business Control Center reporting layer on Hadoop using Hive/Impala to track cardholder rewards liability.
  • SQL-to-Hadoop Transition: Migrated 40+ legacy SQL Server jobs to Hadoop big data platform.
  • CLKB Pipeline: Developed automated Core Loyalty Knowledge Base data mart using PySpark.
  • COVID-19 Response: Built expedited relief reporting to track skipped payments.
  • Git Governance: Established Bitbucket version control standards and CI/CD promotion procedures for the analytics team.
BIM Analyst
TD Bank · Decision Science · Nov 2015–Aug 2017
Click to view achievements
  • Sandbox Solution: Constructed centralized SAS Sandbox environment to streamline campaign targeting logic.
  • Tetris Loyalty LP6: Developed post-campaign measurement dashboards using Tableau and SQL.
Earlier Experience
OGO Fibers / MPR Consulting · 2012–2015
Click to view achievements
  • OGO Fibers (Logistics Coordinator): Built automated VBA macros for shipping schedules and inventory tracking.
  • MPR Consulting (Data Analyst): Analyzed survey response data using SPSS and Excel; optimized data cleaning procedures.

Key Projects & Initiatives

RMR2.0 Lakehouse Framework

Led architectural POC for migrating Regulatory Margin Reporting to Databricks Lakehouse architecture utilizing Unity Catalog for enterprise governance.

UC-enabled AZ PoC validated end-to-end
Databricks Unity Catalog PySpark Power BI

RMR + DAC → Power BI Premium

Spearheaded the complex migration of highly visible executive dashboards from legacy Tableau to modern Power BI Premium workspaces.

18 Dashboards; 100% Tableau decommission
Power BI DAX SQL Server

BISOPS Triage Dashboard

Automated a high-volume data access request process for the business operations team, eliminating manual effort and reducing daily processing time from 2.5 hours to under 30 minutes (80% reduction), significantly improving operational efficiency and scalability.

80% Reduction in Processing Time
Power BI Python/Pandas PySpark

SRZ Synapse Inventory Onboarding

Designed and orchestrated a robust data pipeline to feed critical inventory records into Azure Synapse Analytics SQL pools.

Data availability rate boosted to 99.5%
Azure Synapse ADLS ETL

AI-Augmented Engineering Workflow

Integrated generative AI tools and custom LLM workflows into the team's software development lifecycle to assist with legacy code modernization.

Productivity uplift on code, docs, migration
GitHub Copilot LLM Scripts Python

Enterprise Data Catalog — Purview

Configured and deployed Microsoft Purview to establish automated data scanning, lineage extraction, and PII sensitivity labeling.

Unified discovery + lineage; PII classification
MS Purview Data Lineage Azure

Project Granite — TBSM

Coordinated SIT, parallel run testing, and production cutover for migrating massive retail volumes to target liquidity engines.

Led E2E SIT + FOV production validation
SQL Server Data Migration Data Quality

Merchant Solutions Migration

Developed critical ETL and validation frameworks for migrating a $30B credit card portfolio from First Data to TSYS.

Zero-defect cutover in 6 months
Hadoop SQL Tableau

Certifications & Education

Azure Data Engineer Associate (DP-203)
Microsoft
Azure Data Fundamentals (DP-900)
Microsoft

Academic Background

🎓

M.Sc. Mathematics

Queens College, CUNY

📜

Graduate Certificate

Marketing Research & Analytics

Centennial College

🎓

B. Mgmt, Marketing

Dalian Jiaotong University

Latest Thoughts

The $17 Billion Question: JPMorgan's AI Bet

Exploring how JPMorgan's massive $17B investment is transforming banking tech careers, shifting the focus from coding to AI-augmented solution architecture.

Read Article →

From Prompting to Partnership with AI

A guide on moving beyond basic prompting by utilizing structured frameworks like CO-STAR and CIR to establish strategic communication with AI tools.

Read Article →

From Hallucination to Execution: API Reliability

A developer's guide detailing a 3-stage workflow for productionizing LLM APIs, solving reliability issues by separating AI data extraction from deterministic code rendering.

Read Article →

35+ technical posts on AI, data engineering, and BI solutions.

Visit Blog →

Let's Connect

I'm always open to discussing data engineering challenges, modern lakehouse architectures, or opportunities where data meets AI.

Download Resume