Projects
1. Greenfield - Cloud ETL Pipelines Link to heading
Lead the design and build of complex ETL pipelines in the cloud which delivers the Operational systems data to On-premise Data lake and Data warehouse.
- Technologies: Python, AWS Step functions, AWS Lambda, AWS Eventbridge, SFTP
2. Greenfield - Cloud ETL Pipelines Link to heading
Lead the design and build of complex ETL pipelines in the cloud which serves the data to operational API which is then fed to Machine learning models.
- Technologies: Python, Step functions, Lambda, AWS Eventbridge
3. Egress Framework Link to heading
Lead the design and build of egressing data products from On-premise to cloud.
- Technologies: Python(FastAPI- Server), Scala(Client), AWS ECS, AWS ALB, Kong Gateway, IDP, Grafana(API monitoring), PagerDuty(Alerting)
4. Deal Pricing Express Link to heading
Lead the design and build of ETL pipeline to process Institutional lending data and load it to Data Lake and Warehouse.
- Technologies: Scala Spark, Hive, Autosys, Teamcity, Teradata, Data Modelling(Erwin)
5. Salesforce Marketing Cloud Link to heading
Lead the design and build of ETL pipeline to process Marketing data and load it to Data Lake and Warehouse.
- Technologies: Scala Spark, AWS MSK/Kafka, Abinitio, Hive, Autosys, Teamcity, Teradata, Data Modelling(Erwin)
6. Automated Testing tool for the API Migration to Cloud. Link to heading
Lead the design and build of Automated testing framework for the API migration from On-premise to Cloud.
- Technologies: Python, Teamcity, Mongo, Docker, Teradata
7. Operational Metadata API Uplift Link to heading
Built the REST API endpoint for a Operational Metadata backend for retriving the ETL Load Assurance Metrics.
- Technologies: Scala, Oracle, RESP API
8. 
Link to heading
CommbankIQ: Building ETL pipeline in Spark to process the retail data (de-identification, salt and Hash, Bouncy Castle algo, Vault) for the analysis and transfer to AWS.
- Technologies: Scala Spark, Spark UDF, Boucy Castle Algorithm, Salt & Hash, Scala, Teamcity, AWS S3, AWS Glue, Autosys
9. Alliance Link to heading
Alliance - Building data pipeline to process Intuitional lending data for Group treasury and financial data mart
- Technologies: Python, Unix Scripting
10. Mainframe to Hadoop Migration Link to heading
The Apps Migration project is focused on transferring all claims and enrollment data from Mainframe systems to a Hadoop platform. This migration aims to enable the generation of various reports, such as miscellaneous reports, Adhoc reports, Medical Fraud Detection Reports, GCR, LCR, and Underwriting reports.Redesigning existing systems to leverage Hadoop’s distributed processing capabilities, overcoming the limitations of legacy platforms, and streamlining business processes by removing redundancy.
- Technologies: Mainframe, PySpark, Hive, HDFS, Python, and Unix scripts