View Infrastructure
OmicsPipelines allows you to view certain types of infrastructure related to running pipelines in your chosen cloud provider. Select a cloud provider from the list below to view infrastructions.
- Amazon Web Services
- Google Cloud Platform
Amazon Web Services
Overview
By default, OmicsPipelines only allows you to see infrastructure filtered by region.
Regions
A region is a geographical area that consists of two or more data centers. Each region is completely isolated from the other regions. This means that you can have the same resources in two different regions, and they will not interfere with each other.
AWS CloudFormation
OmicsPipelines leverages a service called CloudFormation provided by Amazon Web Services (AWS) to create and manage infrastructure. CloudFormation allows a user to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.
Why CloudFormation?
CloudFormation allows you to create and manage infrastructure as code. This means that a user can create and manage infrastructure using a text file that describes the resources to be created. This text file is called a template, and functions as a single source of truth for the infrastructure that you want to create. Deviations from the template are tracked and can be rolled back to the last known good state.
What are Stacks?
A stack is a collection of resources that are created as a single unit. A stack can contain all the resources needed to run an application, such as a virtual machine, database, and security rules. A stack can also contain instructions for running a set of services, such as a web application and a database. A user can create a stack from a template that you write, or from a template that someone else has written. A stack template can take in parameters, such as the name of an S3 bucket or the name of a database, and use those parameters to create the resources in the stack.
OmicsPipelines' CloudFormation Templates
OmicsPipelines uses two sets of templates to configure infrastructure, one for infrastructure needed to run pipelines, and the other in order to run the OmicsPipelines dashboard web application.
We call these the Core Environment and the Server.
Core Environment
The Core Environment is the infrastructure needed to run pipelines. It consists of the following resources:
- 1 VPC - This is a virtual network in AWS, similar to setting up Wi-Fi in your home. It allows you to isolate your infrastructure from the rest of the internet.
- 4 Subnets - These are subnetworks within the VPC. They allow you to isolate your infrastructure into different sections, and can be used to control traffic flow between different parts of your infrastructure.
- 2 NAT Gateways - This functions as a one way network traffic valve that allows your infrastructure to communicate with the internet, but not the other way around.
- 1 Batch Queue - This is a queue to which "jobs" may be submitted. These jobs are then run on the compute resources that are attached to the queue.
- 1 Compute Environment - This defines the hardware resources that the Batch Queue's "jobs" are allowed to run on.
- 1 Instance Template - This is a set of instructions that configure the software environment inside each virtual machines provisioned by the Compute Environment.
- 1 CodeCommit Repository - This is a repository in which the code used by the Instance Template to provision Batch Queue's virtual machines is stored.
- 1 CodeBuild Pipeline - This allows the Instance Template to automatically update itself when the user decides to update it (for example, when a new version of OmicsPipelines is released).
- (Optional) 1 S3 Bucket - This is a bucket used to store the templates and artifacts used by the various Core Environment resources, and is not created if the bucket already exists.
- Various IAM Policies and Roles - These are used to control access to the various resources in the Core Environment by all the other resources in the Core Environment.
- Various CloudWatch Log Groups - These are used to store logs from the various resources in the Core Environment, and to allow you (or us if you send them to us) to debug issues with the Core Environment.
Server
The Server is the infrastructure needed to run the OmicsPipelines dashboard web application. It consists of the following resources:
- 1 EC2 Instance - This is the virtual machine on which the OmicsPipelines dashboard web application is run.
- 1 Instance Role - This is the set of permissions that the EC2 Instance uses to access the various resources in the Core Environment.
- 2 Security Groups - This is a set of rules that control the traffic that is allowed to flow into and out of the EC2 Instance. By default, traffic is only allowed to flow into the EC2 Instance from the internet via the SSH port (22).
- 2 CloudWatch Log Groups - These are used to store logs from the various resources in the Server, and to allow you ( or us if you send them to us) to debug issues with the Server.
Change Regions
You can change the region that you are viewing by clicking on the dropdown menu in the top of the page.
Stack View
You can view a list of all the stacks that you have created in the application. A stack can either be a Core Environment or a Server.
Although the button to create a new Core Environment is listed next to the stacks, it is important to note that both Core Environment and Server stacks are created the same way, and this view will show both types of stacks.
Stack Details
You can view details of a stack by clicking on it in the list of stacks. A stack consists of parameters, resources, and outputs, all of which are defined in code. You can view each of these in the stack details modal.
Refreshing the List of Virtual Machines
You can also manually refresh the list of CloudFormation stacks. This is useful, for example, if someone else created a new one, and it doesn't appear in the list.
Virtual Machine View
You can view a list of all the virtual machines that you have created in the application.
Refreshing the List of Virtual Machines
You can also manually refresh the list of virtual machines. This is useful, for example, if someone else created a new one and it doesn't appear in the list.
Viewing the Details of a Virtual Machine
You can view the details of a virtual machine by clicking on the i button of the virtual machine.
You can view the entire JSON document that contains the details of the virtual machine.
Additional actions are also available for the virtual machine, which you can learn about in the Manage Infrastructure page
Google Cloud Platform
Overview
OmicsPipelines provides a single view to see all of your virtual machines in Google Cloud Platform.
This view provides you with a number of functions to manage your virtual machines.
Filtering the List of Virtual Machines by Region
You can filter the list of virtual machines by which region they are located in.
-
Select a region from the drop-down menu (1).
-
Click the "Refresh List of VM Instances in Selected Region" button (2) to update the list of virtual machines.
You can click the button again to refresh the list of virtual machines.
Refreshing the List of Virtual Machines
You can also manually refresh the list of virtual machines, filtered or unfiltered by region. This is useful, for example, if you just created a new one and it doesn't appear in the list.
Viewing the Details of a Virtual Machine
You can view the details of a virtual machine by clicking on the i button of the virtual machine.
You can view the entire JSON document that contains the details of the virtual machine.
Additional actions are also available for the virtual machine, which you can learn about in the Manage Infrastructure page
Next Steps
Now that you have learned how to view infrastructure, learn how to create a new OmicsPipelines dashboard section to learn more.