Projects

1. Greenfield - Cloud ETL Pipelines Link to heading

Lead the design and build of complex ETL pipelines in the cloud which delivers the Operational systems data to On-premise Data lake and Data warehouse.

  • Technologies: Python, AWS Step functions, AWS Lambda, AWS Eventbridge, SFTP

2. Greenfield - Cloud ETL Pipelines Link to heading

Lead the design and build of complex ETL pipelines in the cloud which serves the data to operational API which is then fed to Machine learning models.

  • Technologies: Python, Step functions, Lambda, AWS Eventbridge

3. Egress Framework Link to heading

Lead the design and build of egressing data products from On-premise to cloud.

  • Technologies: Python(FastAPI- Server), Scala(Client), AWS ECS, AWS ALB, Kong Gateway, IDP, Grafana(API monitoring), PagerDuty(Alerting)

4. Deal Pricing Express Link to heading

Lead the design and build of ETL pipeline to process Institutional lending data and load it to Data Lake and Warehouse.

  • Technologies: Scala Spark, Hive, Autosys, Teamcity, Teradata, Data Modelling(Erwin)

5. Salesforce Marketing Cloud Link to heading

Lead the design and build of ETL pipeline to process Marketing data and load it to Data Lake and Warehouse.

  • Technologies: Scala Spark, AWS MSK/Kafka, Abinitio, Hive, Autosys, Teamcity, Teradata, Data Modelling(Erwin)

6. Automated Testing tool for the API Migration to Cloud. Link to heading

Lead the design and build of Automated testing framework for the API migration from On-premise to Cloud.

  • Technologies: Python, Teamcity, Mongo, Docker, Teradata

7. Operational Metadata API Uplift Link to heading

Built the REST API endpoint for a Operational Metadata backend for retriving the ETL Load Assurance Metrics.

  • Technologies: Scala, Oracle, RESP API

8. CommbankIQ Link to heading

CommbankIQ: Building ETL pipeline in Spark to process the retail data (de-identification, salt and Hash, Bouncy Castle algo, Vault) for the analysis and transfer to AWS.

  • Technologies: Scala Spark, Spark UDF, Boucy Castle Algorithm, Salt & Hash, Scala, Teamcity, AWS S3, AWS Glue, Autosys

9. Alliance Link to heading

Alliance - Building data pipeline to process Intuitional lending data for Group treasury and financial data mart

  • Technologies: Python, Unix Scripting

10. Mainframe to Hadoop Migration Link to heading

The Apps Migration project is focused on transferring all claims and enrollment data from Mainframe systems to a Hadoop platform. This migration aims to enable the generation of various reports, such as miscellaneous reports, Adhoc reports, Medical Fraud Detection Reports, GCR, LCR, and Underwriting reports.Redesigning existing systems to leverage Hadoop’s distributed processing capabilities, overcoming the limitations of legacy platforms, and streamlining business processes by removing redundancy.

  • Technologies: Mainframe, PySpark, Hive, HDFS, Python, and Unix scripts

Personal Projects Link to heading

1. Anaca the AI SQL Bot Link to heading

2. PySpark ETL Framework Link to heading

PySpark ETL Framework

3. Redshift Datawarehouse Load - Cloud ETL pipeline Link to heading

Redshift Datawarehouse Load

4. FastAPI on ECS Link to heading

FastAPI on ECS

5. CI/CD AWS Link to heading

CI/CD AWS