Deployment Model

 DataOpsServer

 DataOpsRepository

 DataOpsEngine

 Single-Tier:

In Single-tier architecture, we are installing DataOpsServer and DataOpsRepository and DataOpsEngine on the same machine

Note: Not recommended for production

Processor:32 Cores

Memory: 128GB

Harddisk: 1000GB

 Same as DataOpsServer

 Same as DataOpsServer

 Two-Tier:

In Two-tier architecture, we are installing DataOpsServer and DataOpsRepository in the same machine and DataOpsEngine on different machine

Processor:16 Cores

Memory: 32GB

Harddisk: 500GB

  Same as DataOpsServer

 Three-Tier:

DataOps application consists of mainly three components(DataOpsServer, DataOpsRepository, DataOpsEngine). We are installing these three components in three different machines

Processor:16 Cores

Memory: 16GB

Harddisk: 200GB

Processor:8 Cores

Memory: 16GB

Harddisk: 500GB

Standalone DataOpsEngine

Processor:32 Cores

Memory: 128GB 

YarnCluster with three nodes

Processor:16 Cores(on each node)

Memory: 128GB(on each node)

Harddisk: 1000GB(On each node)

Note: we can scale in or scale out the number of cores and number of nodes based on data

AWS EMR

Processor:8 Cores on master, 16 cores on core and task nodes

Memory: 64GB on master, 128GB on Core and task nodes

Harddisk: 200GB on master, 1000GB on Core and task nodes

Recommended  Type: master r6g.2xlarge,core and task nodes-r6g.4xlarge

Note: we can scale in or scale out the number of cores and number of task nodes based on data

Azure databricks

Processor:16 Cores on worker and 8 Cores on driver

Memory: 128GB on worker and 16GB on driver