The purpose of this post is to provide a high level plan for implementing ETL Validator. 

  • Evaluate Project and System Requirements
    • Understand project testing requirements for Data-in-Motion (data comparison) and Data-at-Rest (data quality)
    • Understand current testing process
    • Prepare a list Data Sources for testing : Relational, Flat Files, Big Data, Cloud, NoSQL, etc.
    • Understand network and system access requirements for the above Data Sources 
    • Estimate data volumes for testing
    • Estimate number of test cases and type of test cases for the initial project delivery
    • Estimate number of users - QA, Development
    • Determine DevOps strategy for CI/CD of ETL and test cases
    • Evaluate Application Lifecycle Management and reporting requirements for test results
  • Evaluate Security Requirements
    • Understand the team and project structure
    • Categorize teams/projects into different groups for separation of tests in ETL Validator
    • Evaluate team access requirements for creating Tests and administering ETL Validator
    • Understand data security requirements between teams
    • Understand ETL Validator deployment requirements for different environments/timezones/regions
  • Installation & Setup
    • Estimate ETL Validator Server and Workschema hardware sizing based on Data Volumes and number of users/tests
    • Estimate hardware for ETL Validator Servers and Remote Desktops for ETL Validator Clients based on the number of environments/regions
    • Procure and setup hardware for ETL Validator Server in a network location closer to data sources
    • Configure network access for server machine to Data Sources (open ports)
    • Verify access to Data Sources from the server machine
    • Install ETL Validator Server and Repository/Workschema Database (PostgreSQL or Oracle)
    • Setup multiple Workschemas based on data security requirements between teams
    • Install ETL Validator Client and generate a Repository access file
    • Create Data Source connections and test them
    • Setup ETL Validator CLI tool for CI/CD integration
    • Setup a backup/recovery process for ETL Validator Repository/Workschema
    • Configure SMTP settings for Email Notification
  • Provision & Train Users
    • Setup Containers, Groups based on the team/project requirements
    • Setup Workschema and Data Source connections as per the team/project requirements
    • Provision users to the appropriate ETL Validator groups (and containers)
    • Provide users access to install ETL Validator Client and import the Repository access file
    • Train users on ETL Validator : Videos, Use Cases, Instructor Led
  • Plan, Create and Execute Tests
    • Understand different types of tests in ETL Validator and come up with a plan for creating tests
    • Create tests based on use cases, user stories as part of your Project Plan
    • Use parameters for reducing the changes to test cases
    • Group Tests into test plans for executing them together
    • Setup notifications for test plans
    • Schedule test data purge if needed
    • Schedule test plan execution
    • Automate test case and test plan runs using ETL Validator CLI tool as part of your CI/CD process