Practice Design for Try/Fail Fast

Batch Processing, Parallel Processing

• Batch Processing: allows users to submit series of programs (jobs) and they will be executed to completion without further user input and manual intervention. Its popular usage to separate large work processes into sub small jobs such as build reports, big data processing. And the batch jobs can parallel running with distributed computing and implementation at architecture.
• Parallel Processing: is the processing of program instructions by dividing them among multiple processors to use the CPU power of a computer to run a program in less time and implement programming.

Batch and parallel are popular solutions to use the power of a computer to improve the performance of a program. So, according to the requirement and application of the solution that we can choose the fit solution. Batch processing stronger than parallel processing at distributed computing. Parallel processing optimal on single machine and batch processing usage for larger workload with distributed computing.

You can go to Apache Spark, Hadoop or AWS Map Reduce to learning more about batch processing and optimization. With CI/CD, the runner machines can host with multiple mechanisms belong specific jobs. I like to use Gitlab CI with custom runners. I can host specific jobs to specific machines to separate the major workload. More information about Gitlab Runner auto-scaling with AWS Fargate

Thanks for reading,
Please give me comments if you have any ideas and suggestions. I hope to learn more from you.

References:
– Microservice architecture document: https://microservices.io/
– AWS Microservice deployment with Fargate: https://d1.awsstatic.com/whitepapers/microservices-on-aws.pdf