Copyright (c) Mizuho Information & Research Institute, Inc. All Rights Reserved.

Transkript

1 Copyright (c) Mizuho Information & Research Institute, Inc. All Rights Reserved.

2 n < > AI n 2018 AIPoC p p

3 n n AWS n Kubernetes n n n CD

4

5 n n DB API API API AP SPA Data Center Cloud Legacy Modern

6 n Docker Kubernetes n AWS n AWS KubernetesEKS(Elastic Container Service for Kubernetes) KOPS n Helm

7

8 Feed Agent Convert Streaming CD Data Sources Visualization Realtime Processing x

9 Feed Agent Convert Streaming CD Data Sources Visualization n n Realtime Processing x

10 Feed Agent Convert Streaming CD Data Sources Visualization Realtime Processing n p p x

11 Data Sources Feed Agent Convert Streaming n n n CD Visualization Realtime Processing x

12 Data Sources Feed Agent Convert Streaming n n CD Visualization Realtime Processing x

13 n Data Sources Feed Agent Convert Streaming RabbitMQ CD Visualization Realtime Processing x

14 Feed Agent Convert Streaming CD Data Sources Visualization Realtime Processing n S3 x

15 Feed Agent Convert Streaming CD Data Sources n n Visualization Jupyter Notebook Realtime Processing x

16 Feed Agent Convert Streaming CD Data Sources Visualization n Spinnaker CD Realtime Processing x

17 AWS

18 VPC Availability zone 1a Internet gateway Availability zone 1c Public Subnet Public Subnet NAT gateway NAT gateway Private Subnet Private Subnet Router Router

19 VPC Availability zone 1a Internet gateway Availability zone 1c Public Subnet Public Subnet NAT gateway NAT gateway Private Subnet Private Subnet Router Router

20 Kubernetes

21 KOPS n Kubernetes AWSk8s n GCP β n EC2 LoadBalancer AWS n Terraform n S3

22 KOPS n Kubernetes $ kops create cluster ${NAME} --master-zones ap-northeast-1a,ap-northeast-1c --zones ap-northeast-1a,ap-northeast-1c --vpc vpc-xxxxxxxxxx --topology private --master-count=3 --master-size t2.xlarge --node-count=0 --target=terraform --out=.

23 KOPS n Master Node AZ $ kops create cluster ${NAME} --master-zones ap-northeast-1a,ap-northeast-1c --zones ap-northeast-1a,ap-northeast-1c --vpc vpc-xxxxxxxxxx --topology private --master-count=3 --master-size t2.xlarge --node-count=0 --target=terraform --out=.

24 KOPS n VPC ID $ kops create cluster ${NAME} --master-zones ap-northeast-1a,ap-northeast-1c --zones ap-northeast-1a,ap-northeast-1c --vpc vpc-xxxxxxxxxx --topology private --master-count=3 --master-size t2.xlarge --node-count=0 --target=terraform --out=.

25 KOPS n Worker Node Private Subnet $ kops create cluster ${NAME} --master-zones ap-northeast-1a,ap-northeast-1c --zones ap-northeast-1a,ap-northeast-1c --vpc vpc-xxxxxxxxxx --topology private --master-count=3 --master-size t2.xlarge --node-count=0 --target=terraform --out=.

26 KOPS n Master Node $ kops create cluster ${NAME} --master-zones ap-northeast-1a,ap-northeast-1c --zones ap-northeast-1a,ap-northeast-1c --vpc vpc-xxxxxxxxxx --topology private --master-count=3 --master-size t2.xlarge --node-count=0 --target=terraform --out=.

27 KOPS n Worker Node p 0 Worker Node $ kops create cluster ${NAME} --master-zones ap-northeast-1a,ap-northeast-1c --zones ap-northeast-1a,ap-northeast-1c --vpc vpc-xxxxxxxxxx --topology private --master-count=3 --master-size t2.xlarge --node-count=0 --target=terraform --out=.

28 KOPS n KOPSterraform $ kops create cluster ${NAME} --master-zones ap-northeast-1a,ap-northeast-1c --zones ap-northeast-1a,ap-northeast-1c --vpc vpc-xxxxxxxxxx --topology private --master-count=3 --master-size t2.xlarge --node-count=0 --target=terraform --out=.

29 n n 3

30 apiversion: kops/v1alpha2 kind: InstanceGroup metadata: name: compute-nodes-1a spec: image: kope.io/k8s-1.11-debian-stretch-amd64-hvm-ebs machinetype: t2.xlarge maxsize: 2 minsize: 1 nodelabels: instance: t resource: cpu subnets: - ap-northeast-1a - ap-northeast-1c

31 n EC2 apiversion: kops/v1alpha2 kind: InstanceGroup metadata: name: compute-nodes-1a spec: image: kope.io/k8s-1.11-debian-stretch-amd64-hvm-ebs machinetype: t2.xlarge maxsize: 2 minsize: 1 nodelabels: instance: t resource: cpu subnets: - ap-northeast-1a - ap-northeast-1c

32 n p apiversion: kops/v1alpha2 kind: InstanceGroup metadata: name: compute-nodes-1a spec: image: kope.io/k8s-1.11-debian-stretch-amd64-hvm-ebs machinetype: t2.xlarge maxsize: 2 minsize: 1 nodelabels: instance: t resource: cpu subnets: - ap-northeast-1a - ap-northeast-1c

33 n Worker Node PodNodeSelector apiversion: kops/v1alpha2 kind: InstanceGroup metadata: name: compute-nodes-1a spec: image: kope.io/k8s-1.11-debian-stretch-amd64-hvm-ebs machinetype: t2.xlarge maxsize: 2 minsize: 1 nodelabels: instance: t resource: cpu subnets: - ap-northeast-1a - ap-northeast-1c

34 n Subnet apiversion: kops/v1alpha2 kind: InstanceGroup metadata: name: compute-nodes-1a spec: image: kope.io/k8s-1.11-debian-stretch-amd64-hvm-ebs machinetype: t2.xlarge maxsize: 2 minsize: 1 nodelabels: instance: t resource: cpu subnets: - ap-northeast-1a - ap-northeast-1c

35 KOPS n yaml Terraform n Terraform $ kops update cluster --target=terraform --out=. ${NAME} n n Pod $ kops update cluster ${NAME} --yes $ kops rolling-update cluster ${NAME} yes

36

37 Feed Agent Convert Streaming CI/CD Data Sources Visualization Realtime Processing x

38 n Spark n Kubernetes

39 Spark on Kubernetes n kubectl n driver Executor Pod n Client spark-submit spark driverdriverexecutor Clinet spark-submit apiserver scheduler Spark Driver (node A) Spark Executor 1 (node A) Spark Executor 2 (node B) Spark Executor 3 (node B)

40 Spark on Kubernetes n spark submit $ bin/spark-submit --master k8s:// --deploy-mode cluster --conf spark.executor.instances=5 --conf spark.kubernetes.container.image=<spark-image> local:///path/to/examples.jar Clinet spark-submit apiserver scheduler Spark Driver (node A) Spark Executor 1 (node A) Spark Executor 2 (node B) Spark Executor 3 (node B)

41 Spark on Kubernetes n CD spark-submitpod n spark-submit Pod driver Pod yaml p args spark-submitpodspark-submit p lifecycle.prestop driver executor Pod kubectl spark-submit

42 Spark on Kubernetes n spark-submit executor pod --conf spark.executor.instances= 2 --conf spark.executor.memory= 10G --conf spark.executor.cores =5 EFS(Amazon Elastic File System) --conf spark.kubernetes.driver.volumes.persistentvolumeclaim.checkpointpvc.mount.path = fs-xxxxxxxx.efs.ap-northeast-1.amazonaws.com NodeSelector Pod Node --conf spark.kubernetes.node.selector.resource=cpu --conf spark.kubernetes.node.selector.instance=c --conf spark.kubernetes.node.selector.usage=prediction

43

44 Feed Agent Convert Streaming CI/CD Data Sources Visualization Realtime Processing x

45 n Jupyter Notebook n Jupyter Notebook JupyterHub n CPU GPU

46 Jupyterhub with Kubernetes

47 Jupyterhub with Kubernetes n JupyterHub k8s n Helm n Kubespawner Pod(Jupyter Notebook) Persistent Volume Persistent Volume Claim n n 30 Pod

48 Jupyterhub with Kubernetes n CPU GPU p Kubespawner p

49 Jupyterhub with Kubernetes c.kubespawner.profile_list = [ { 'display_name': 'CPU notebook', 'default': True, 'kubespawner_override': { 'singleuser_image_spec': XXXXXXXXXXX.dkr.ecr.ap-northeast-1.amazonaws.com/pyspark-notebook-cpu:latest', 'cpu_limit': 1, 'mem_limit': '3G', 'node_selector': { "resource": "cpu", "instance": "c", "usage": "jupyter" } } }, { 'display_name': 'GPU notebook', 'kubespawner_override': { 'singleuser_image_spec': ' XXXXXXXXXXX.dkr.ecr.ap-northeast-1.amazonaws.com/pyspark-notebook-gpu:latest', 'cpu_limit': 1, 'mem_limit': '8G', 'extra_resource_guarantees': {"nvidia.com/gpu": "1"}, 'extra_resource_limits': {"nvidia.com/gpu": "1"}, 'node_selector': { "resource": gpu", } }]

50 Jupyterhub with Kubernetes c.kubespawner.profile_list = [ { 'display_name': 'CPU notebook', 'default': True, 'kubespawner_override': { 'singleuser_image_spec': XXXXXXXXXXX.dkr.ecr.ap-northeast-1.amazonaws.com/pyspark-notebook-cpu:latest', 'cpu_limit': 1, 'mem_limit': '3G', 'node_selector': { "resource": "cpu", "instance": "c", "usage": "jupyter" } } }, { 'display_name': 'GPU notebook', 'kubespawner_override': { 'singleuser_image_spec': ' XXXXXXXXXXX.dkr.ecr.ap-northeast-1.amazonaws.com/pyspark-notebook-gpu:latest', 'cpu_limit': 1, 'mem_limit': '8G', 'extra_resource_guarantees': {"nvidia.com/gpu": "1"}, 'extra_resource_limits': {"nvidia.com/gpu": "1"}, 'node_selector': { "resource": gpu", } }]

51 Jupyterhub with Kubernetes c.kubespawner.profile_list = [ { 'display_name': 'CPU notebook', 'default': True, 'kubespawner_override': { 'singleuser_image_spec': XXXXXXXXXXX.dkr.ecr.ap-northeast-1.amazonaws.com/pyspark-notebook-cpu:latest', 'cpu_limit': 1, 'mem_limit': '3G', 'node_selector': { "resource": "cpu", "instance": "c", "usage": "jupyter" } } }, { 'display_name': 'GPU notebook', 'kubespawner_override': { JupyterNotebook Docker 'singleuser_image_spec': ' XXXXXXXXXXX.dkr.ecr.ap-northeast-1.amazonaws.com/pyspark-notebook-gpu:latest', 'cpu_limit': 1, 'mem_limit': '8G', 'extra_resource_guarantees': {"nvidia.com/gpu": "1"}, 'extra_resource_limits': {"nvidia.com/gpu": "1"}, 'node_selector': { "resource": gpu", } }]

52 Jupyterhub with Kubernetes c.kubespawner.profile_list = [ { 'display_name': 'CPU notebook', 'default': True, 'kubespawner_override': { 'singleuser_image_spec': XXXXXXXXXXX.dkr.ecr.ap-northeast-1.amazonaws.com/pyspark-notebook-cpu:latest', 'cpu_limit': 1, 'mem_limit': '3G', 'node_selector': { "resource": "cpu", "instance": "c", "usage": "jupyter" } } }, { 'display_name': 'GPU notebook', 'kubespawner_override': { NodeSelectorNode 'singleuser_image_spec': ' XXXXXXXXXXX.dkr.ecr.ap-northeast-1.amazonaws.com/pyspark-notebook-gpu:latest', 'cpu_limit': 1, 'mem_limit': '8G', 'extra_resource_guarantees': {"nvidia.com/gpu": "1"}, 'extra_resource_limits': {"nvidia.com/gpu": "1"}, 'node_selector': { "resource": gpu", } }]

53 CD

54 Feed Agent Convert Streaming CD Data Sources Visualization Realtime Processing x

55 n CodeCommitCodeCommitpush n Docker Build CodeBuild p yaml Buildspec n ECR Docker Helm chart k8s

56 CD n Spinnaker p Netflix CD p Helm p u Build(Webhook) p Webhook CodeBuild u Bake(Helm) p Helm Chart base64 u Deploy p Helm Chart k8s

57 Pipeline POST Build GET Status Succeeded Deploy Status boto3 boto3 S3 Helm Chart k8s push Trigger Push Helm Chart

58 n KOPS AWSKubernetes n Spark On Kubernetes n Jupyterhub with KubernetesJupyterNotebook n Spinnaker CD n

59 n Master Node p EKS n p kiam kube2iam Pod IAM Role n p EKSCloudWatch Container Insights n GPU Jupyter Notebook

60 n Spinnaker AWS CodeCommit CodeBuild Lambda p CodePipeline p Spinnaker n Docker Image

61