This process has 4 tasks
1. Postgres Creation
2. Querybuilder app service Creation
3.DataOpsServer app service Creation
4.DataOps Engine Creation
Assuming you have a Private network setup, these components are created in a private network. To access these endpoints, we need private access to Azure Vnet. If not, we have to use a public endpoint setup.
Azure Postgres flexible server Creation
1a. We have to create a PostgreSQL Flexible server with minimum configuration B4ms. In the below screenshot, we have to select Subscription, create, or select resource-group, input server name, Region,version16, and workload production, If required select high availability, select PostgreSQL and Microsoft authentication then proceed to the network section.
1b. Select private access, subscription, and Vnet, and create or select a dedicated PostgreSQL subnet, If required we can change the options in security and tags otherwise we can go ahead with default options. then click on review and create.
1c.Once the service is started we have to go to settings-->serverparametes --> extensions then select PGCRYPTO, PLPGSQL, UUID-OSSP and save.
1d. Go to the overview of Postgresqlserver and copy the server name we need to give in PG_HOST.
1e. Go to settings -->databases create a database with name "dataflow",we have to use this in PG_DB. finally we have to collect PG_HOST,PG_USER,PG_PASSWORD,PG_DB
2. Deploying QueryBuilder Service
2a. Go to appservice click on create web app select subscription and resource group, give name, publish as container, operating system linux, Region, pricing plan Premium V3 P0V3, and Disable redundancy. then go to the container section.
2b. Select other container registries access type public and image is datagaps/querybuilder:2024.5.1.0
2c. in networking section, enable public access off, enable network injection on, select virtual network, enable private endpoints give private endpoint name, and select or create an inbound subnet. enable Vnet integration select or create an outbound subnet.
2d. remaining steps we can leave for default and proceed to review and create.
2e. after creating querybuilder service we have to navigate to overview screen and copy the default domain. we have to use this as QUERYBUILDER_URL
2f. After the creation of querybuilder service go to settings environmentvariables add Variables which we collected from step 1. PG_HOST,PG_DB,PG_USER,PG_PASSWORD. then save and restart.
3. Deploying DataOpsServer in Azure AppService
3a. go to app services and create webapp select subscription, resource group, give name, publish as a container, operating system linux, region, select minimum plan Premium V3 P2V3, disable redundancy then go to the next step.
3b.select other container registries, access type public, image is datagaps/dataopsserver:2024.5.1.0
3c.In the network section, select Enable public if you don't have private access otherwise you can make it off, and enable network injection on, select Vnet, enable private endpoints, give a private endpoint name, and select Inbound subnet. enable Vnet integration on and select an outbound subnet
3d. Remaining sections we can keep the default, and proceed to create
3e.
After the creation of Dataopsserver service go to settings environmentvariables add Variables which we collected from step 1 and 2. PG_HOST,PG_DB,PG_USER,PG_PASSWORD,QUERYBUILDER_URL. and WEBSITES_CONTAINER_START_TIME_LIMIT=900 then save and restart
3f.After 10min we can access DataOpsServer service endpoint from host which has private access to Azure Vnet.
exampleurl: dataopsserver-drh6cdddfafvfwc3.centralindia-01.azurewebsites.net
ignore any error popups, click on Manage Repository, give username and password of Postgres repository, select upgrade repository then proceed to next steps and upgrade then download the log for future reference in case of any errors.
After this step restart both querybuilder and dataopsserver services, wait for 10minutes then we can open the server endpoint.
4. Deploying DataOpsEngine in AppService.
4a.We can follow the same process as DataOpsServer deployment, except container image datagaps/dataopsengine:2024.5.1.0 and Pricing Plan V3 P3V3 and environment variables PORT=8998,CLUSTER=standalone