This week we went on a small adventure discovering the K8s control plane components as well as the node resource. We shared background, experience, and as usual - went too far with stories ;)
Links:
Meir's blog: https://meirg.co.il
Omer's blog: https://omerxx.com
Telegram channel: https://t.me/espressops
This week we went on a small adventure discovering the K8s control plane components as well as the node resource. We shared background, experience, and as usual - went too far with stories ;)
Links:
Meir's blog: https://meirg.co.il
Omer's blog: https://omerxx.com
Telegram channel: https://t.me/espressops
don't Oh my lady, now I'm ready. Yes. Okay, and action Action, I think I won. Yeah. Yeah. Okay. Yours was definitely more like of a yeah All right, so I see the videos. Okay, so like the last chapter that we had like the last episode of the videos of now Not so in sync. I hope Riverside will fix it, but I think you can hear me loud and clear. Let's pray to the podcast gods Okay, okay, so everything is fine. So hello everyone, and welcome to DevOps topics with a K The instead of a C in the topics. Okay, okay, so we are in the 24th episode and today we are going to talk about yeah 124 not 28 24 so in today we are going to talk about Kubernetes components and when I say components, I mean the Kubernetes components of the Like a control plane, right? So email What is the first thing that comes up to your mind when I say Kubernetes components Okay, I must say let's give a little bit of a background. I'll be honest I thought of this topic this week because I had the pleasure of running into The scheduler I'm building something and I had to kind of mess a little bit with the scheduler understand how it's working What the algorithm behind it will I'll touch in in second what the scheduler actually does But through that we built on we kind of built on that an entire topic and we'll discuss the different components within Kubernetes So specifically the first thing that comes into my mind Purely because of what I've done this week is the Kubernetes scheduler The Kubernetes scheduler specifically is a part that's responsible for Obviously scheduling stuff within the cluster when we say stuff I've deployed a pod and I would need to run somewhere right within one of the nodes scheduler is the ones that responsible For doing that magic It doesn't have to only deal with pods. It can deal with the persistent volumes volume claims, etc There are lots of other components the next thing the immediate next thing Which is not part of the control plane by the way If the component that runs within the the node because that's something I also had to tweak with this week and that's cubelet and cubelet is the Kubernetes agent That's that's running and it's basically instantiating Storage and containers on the on the node itself, etc. So that's the first thing. What's yours? Okay, okay So I'll leave mine alone for a second. I want to reorganize the things you've said So let's divide it like dividing conquer. Okay, so each topic. We will conquer each topic. All right, so Which components components are running on the node workers? The ones that are running in the workloads deposit everything which which planets components? Can you name them? Yes, again, the first first and foremost the main one is cubelet cubelet is the agent I'm pretty sure you don't have the way To replace it unless you replace the entire version of Kubernetes. Please correct me if I'm wrong What you can do with cubelet is provide a lot of flags when you run it so that the when you instantiate a node cubelet starts It's work. It's connecting to the cluster. You can profit a lot of flags to tweak It's method and how it operates and and how it you know Prefer's stuff how it sorts its jobs, etc The other one is cube proxy and cube proxy is the one that's kind of Managing the traffic in and out that's coming in and out to the node So it kind of applies the firewall rules the ports the address, etc I think those are the main two components. Those are okay, and now let's move on to the control plane So those two components are for the agents for the node. That's always yes Walker knows now. What are we having the control plane? You said something about this schedule? So maybe let's let's start it in an order. Maybe let's talk about what's the API cell though? Cube API server. What is that? I think we can even start a little bit above that Basically Kubernetes is is a bunch of resources that I'm thinking on how to say it in a pretty way. You know, let's let's put it in a better way I've one heard Kelsey hi tower describes that Kubernetes is a database Right, you can think of it as that. Why is it that they're based because it's basically A system that holds information of how things should look like. What do I mean by that? Kubernetes can be an and let's say you have an existing system that has a few pods running within it a few pods a few nodes Whatever its Kubernetes running and now you want to make a change The way that you may are making a change is not by speaking to a specific component What you will do is create a resource and just push it into the cluster, right? You'll you'll describe a YAML of the pod It's the most either, right? Obviously you can do that with the CLI But the typical way is creating a YAML of the pod and just crash running whatever kubectl apply Right, assuming that's the way you work and then Kubernetes sees a request for a pod It's a new object So you can kind of think of adding it as an entry to the database the Kubernetes database And there's a system of controllers and that's what I wanted to say to begin with So there are a bunch of different controllers each one of them is responsible for a different set of resources So for example This would be the controller that's in charge of pods and that would respond to a new event that there's a new Resource called kind pod within the system and it will start iterating over it The way does that I don't know if people care all that much you said something about the database So let's shove another component in there So what is this database made of which service is that? So the real database within Kubernetes is ETCD ETCD is a very distributed key value store. That's very easy to manage I think that's why the Kubernetes team went with it. I just wanted to finish the thing about What the controller just so we understand how things work when we respond or Other how Kubernetes responds to new resources The controller runs within a loop It's called a reconciliation loop and that's an important word because that's what it's trying to do That's why the notion of just hang on a sec. I cannot see you anymore. You can't see me No Don't know what to do Do you know what to do with your microphone I'm trying to change my camp if you want you think you see yourself Yeah, I see both of us Maybe we just continue and see if anything is okay Okay, go ahead. Okay Where are the reconciliation? It's ETCD. Yeah, so okay before ETCD the reconciliation loop every controller has a loop It runs through a new event So let's say we have a new event of a pod and now Kubernetes is trying to do something This something is going to be putting the pod to work making it run They are putting getting all the instruction that may provided. Let's say mayor said that he wants an engine x running I Don't know and port 80 open and responding to whatever HTTP Kubernetes will try to create all of these so the controller will iterate through a loop and will try to Make that work if it doesn't work for some reason. Maybe the node is not ready Maybe we don't have enough space. Maybe the node is restarting Maybe I don't know what how can it know on the node? Like how can like how does it communicate with a node and then it knows if it works or not? What is it seeking in the node? Okay, so that's how a scheduler works So before I'm going to jump in there We were opening a lot of the parentheses here So just to finish that as long as it doesn't Make something work and there's still something to wait for it will not wait Per se it will just push it again to the loop right So you have an infinite loop running on events and that's how Kubernetes basically works the entire thing It's a bunch of controllers The thing that manages these controllers is called the controller manager So it's kind of a run time that runs within the Kubernetes Control plane and it just runs a bunch of containers responding to events now to your question you asked How do we basically choose where the prot runs or how do I mean you said I'm trying to reach a node and it's not available anymore How is it trying to reach the node and then actually what I'm trying to To talk about is again the cube let's so everything when you try to talk with the node I'm assuming you're talking like I'm trying to communicate with a node and according to its cube Let's agent. I'm getting information about that node and it's running Pods and containers and whatever correct exactly exactly that so okay. Let's talk about that specifically Let's say we're still in the same example. We're trying to run an engine export What's happening is Kubernetes first of all it wants to know Which note it needs an answer? Where am I going to deploy it? And when we say Kubernetes in that sense that's a controller that's responsible for the pod So it waits for an annotation coming from the cube scheduler The cube scheduler is just another controller running on a loop of events It sees a pod and this pod says it's pending. It's not running So the scheduler understands that it's something that it needs to handle it needs to provide the scheduling to How does it do that and that's the interesting part that I've learned this week It has two basic things that it needs to make sure One is filtrate. It's basically a filter, right? It goes through a bunch of nodes Let's say you have five nodes, okay, and let's say your pod needs Two CPUs and four gigabytes of memory First of all it'll go through the entire set of nodes and see where does it have these available resources to it If let's say three of these nodes are already taken with space because of other pods It'll just eliminate them from the list. How do I know what if I'm a rookie and I didn't declare any requests or limits or anything on my pod You just get the basics do you remember the defaults? I think so the first thing goes through something about All right, so resources is the first thing you do and if you don't you if you don't declare them Then it can just go It's keeps like that step or something because it sounds weird, you know If you like if I don't declare the resources request and limits, how will it choose a node? Let's say all the nodes are the same they have the same annotations. I know ladies sending a question back to you Can you not declare at all resources? I don't think you can for sure in deployments for sure. Yeah, yeah, it's fun Because you never know what's gonna happen Yeah So when you do horizontal pod auto scaling for example you must declare Real requests just as an example because the horizontal pod auto scaling depends on Request otherwise you cannot use the metric server and whatever and it will get to that But if you don't declare Anything then maybe the step that you're talking about is skipped I'm not sure maybe we'll reading the docs about it, but assuming we declare the requests and limits I guess I guess what's happening under the hood is that it is assigned with something because it needs like a basic There's probably a basic slot, right? If you we go to the lowest values of each is I think it's a quarter of a CPU and Something like half a gigabyte or something like anyway No, it's so yes, that would be part of a filter. That's actually an interesting spot to check It filters everything up nodes that are not a good job interview question. Yeah Let's keep it So again filter Shorter list now. We have two nodes that can match the resources that we need and now Kubernetes goes to those are not Hard coded limits anymore now. It has like preferences. Let's call it So we have Thanes and affinity and and selectors label selectors and stuff like that and then it's it can kind of sort the list in terms of preference Where would I like to be deployed and maybe I'd like to be deployed in a machine with a GPU Maybe I like to be deployed in a machine that has a specific label and it will sort them Just like a sorted list. It will pick the first one and that's how the scheduler works Once he says that it provides a scheduling and everything else can happen The else just in a setting just in one sentence is telling kublet On the specific note that it was scheduled, please run that container and then kublet does its thing Docker run well not Docker anymore, but yeah, just start the container and where does the Cube API server server takes place like what is that in this whole communication stuff where it's gonna scale down scale up What's the API server for uh, okay, so The API server can kind of interact between different components within the cluster So you do actually it's not really a database you do have to Communicate between different components and that's the way you do that So the API server exposes that so for example if I run the Dashboard right we have the UI the default UI that comes with Kubernetes If you run the Kubernetes dashboard it needs endpoints like API endpoints to interact with to understand how many pods I have how many pvc's I have Where things deployed what is going on with the notes that's the API server when you run a kubectl command That speaks to the API server and fetches the information unit All right, all right Okay, so let's move on to more Kubernetes components so far. We covered that at cv is the database Right, yeah, cube schedule is for assigning pods Not only pods, but mainly us Okay, and we talked about the kubectl for exposing the Kubernetes API to kubectl help or any other API command that want to run maybe Not in when not in the cluster like as a administrator since cic in whatever Right cube controller manager We also talked about that which includes the like a lot of other controllers like replication controller So there's a counter it contains many controllers and what does it do remind me you talked about maybe a little consiling or something? So like The controller manager talking about right the controller manager is in charge of the controller run time So that it makes sure that controllers are alive Our controllers have this system where they live in so that they have the notion of a leader and a follower And it makes sure that if the leader is dead and the follower takes over and it's it's built in a highly scalable way So it's a highly available So that's what it does And we also have the kublet and the kuboxy Well, both of them are Running on the nodes and according to them things are like we can communicate with the nodes and get traffic into the nodes And run pods containers or whatever like any workload on the nodes So we covered all of those Okay, all of those components, okay So what's so special about the cube scheduler? When we started this conversation you sound excited about whom cube scheduler. What's so exciting about that? May I stop to hear you for a few seconds? Can you maybe check the microphone that it's all okay? Is it okay now feels okay? Yes I don't know if any if you did anything He didn't touch I just did with my hands. Yeah, now I hear it perfect. Okay. It was very I only heard it in one ear Maybe it's human beats him Okay, so What's so special in cube schedule? Okay, what's so special about the cube scheduler? Kabler's scheduler is not all that special What's interesting is what we've discussed earlier about the algorithm behind it and the algorithm behind it has Like we won't go into specifically what it does But yes, it's built on top of filtering notes that are irrelevant and then Signing them in a sorted list based on preference of what we've configured to the pod for example It has another notion which sometimes it needs to provide scheduling even before things are happening for example Something I had to do this week when you assign a PVC A persistent volume claim. You're basically telling Kubernetes. Please create a type of storage if you the listener are not aware Kubernetes is slowly ditching all the what we used to call entry Entry CSI's because you used to be able to tell Kubernetes I want an AWS CBS and Kubernetes was just no way it is But then we have a lot of cloud providers today in a lovely way to create storage So Kubernetes is slowly taking everything out and it's telling the providers you create the layer This layer is called the CSI If you are running a PVC today and in the modern CSI There's an EBS CSI for Amazon for example, you can put any CSI for any cloud provider or any provider at all The claim basically tells the cloud. I need a volume. Please give me a pv. That's surrounding a volume But you can tell the cluster. Please don't do it immediately I don't need the persistent volume now. I'm just describing something that I will need later So there's a storage class and I hope I'm not going too deep But there's a storage class that you can refer to on top of each volume And you can tell that please use that storage class and the storage class can say Don't create anything. Please wait for a first consumer. That's the actual notion Wait for a first consumer and the reason is we can create Just informational resources within Kubernetes to make it a lot more efficient And then when someone needs it when I have a consumer that actually needs that PVC then it goes in provisions and the volume If you do that the scheduler will need to provide a scheduling even before things are happening Right, there's a PVC ready. It's waiting to be created somewhere And even before everything happened, there's no consumer. The scheduler will provide a scheduling And it will go through the same thing. It will go through the filtering so and the sorting and everything else And it will provide the best node for the PVC to live even if it's not yet there. So that's the The interesting magic that I've learned this week I also want to give an example now that you mentioned it sounds interesting because Also how the way ingures controller work It sounds the same when you're talking about the first consumer So we need to deploy an NG next for example NG next in this controller Maybe the ALB controller in AWS. You know the application of bands are controlling the AWS So you deploy the controller, but when only after you create an ingress You know the ingress object you actually create a listener So it's funny how when you know one concept of like when you said the first consumer or a consumer like you're like, okay This is only the allocation. It might be Implemented or not, but it's on demand. So I like it. You know that everything is the same. So it's also ingress also in volumes Yes, I think that's a it's a concept in Kubernetes right being able to do everything It used to be that's why Kubernetes got a lot of hate in the past because people would say and that's not only Kubernetes Any other system if it can do everything you get lost in the woods, right? Usually as engineers we do want to be able to tweak stuff But if you get a system that tells you it's completely opinion I didn't there's only one way to do a thing you as an engineer would probably be a little bit frustrated with it other people Probably not so much So I think Kubernetes is very extreme in that sense. You can basically do anything you can Almost I mean the fact that you have controllers and you can write your own controllers and replace anything You can basically do whatever you want the fact that Kubernetes can now decide. Okay, we don't know anymore what storage is You as a provider you create a CSI install whatever CSI you want. We don't care We'll take care of scheduling. We can manage. No, you can manage the idea of storage But we won't take care of it for you just create something on your own. So It's a lot of a lot of work has to come from third parties and a lot of work can come from anyone who wants you can extend it and change Any part of it however you want obviously the project itself is completely open source So you can also fork it not that I think you should but you can fork it and change whatever you want and run your own Mail Kubernetes Very good. Very nice brand. It sounds it sounds like Kubernetes is like the most I Say if I'd say it in a sentence is like the most Flexible best State management system You know like it has all the features that you need To handle states now. It's funny. I'm not gonna dive deep dive into any other Infrastructures code, but if you think about it terraform is infrastructures code So it's also a state management system also pull home your whatever infrastructures code you use But actually you can do the same With Kubernetes because it's not only related to pods and nodes if you create maybe a CRD You know a customer is of definition Let's say an EC2 and I'm creating an EC2 which is a virtual machine as a CRD Maybe it's already you know already exists. I think to give a background of what a CRD is So so a CRD a customer is of definition is like if I want to define something anything that is not natively in Kubernetes So I just can define it with a customer is of definition Following that I can create objects. So let's say my CRD is mail EC2 because this is the type of object I want to create Following that I can create an object with kind May or EC2 because it's defined right and then this object can be anything that I want and it can do anything that I want Okay, so it can be a state management system for anything not just containers not just the cloud You can also manage maybe cues in there. Maybe well, you know line observations or tickets or anything you want state management I wonder if it's the system is gonna you know evolve more than that you know move from containers to manage anything You know, what do you think about that? Would it happen in the future? First of all, I'm smiling because I'm trying to remember a project that basically did something like that They kind of two Kubernetes, but used it only for state management of something. I can't remember what I'll try to find it That's that's one thing. Yeah, so it's super interesting that you've said it And the other one well, yes, I think Kubernetes is trying to be or people are trying to make it a platform or of its own Because I mean, let's say we're both AWS users today, right? But it's not specific for AWS I will use AWS for a lot of my services Databases I'll use maybe rds. I'll take cash mechanism Like elasticish maybe I want to see the end and I'm not maybe not cloud flare I just use Amazon as it can provide a lot of services The work people are trying to do and I see a lot of companies do that nowadays They just install everything within Kubernetes. So I just help install my postgres And I'll help install Obviously elastic and I will install a hell of install read this etc etc And it can be blown up to to not only running one pod you can actually install a cluster of things and they'll come There's also always a fight where like how do you manage the infrastructures Kubernetes? So I know that most organizations were a lot of organizations that I know do it with terraform You know all the VPC the EKS cluster and everything But the workloads are managed in Kubernetes, right? With hell actually for some you manage them So now one part is managed by terraform and then you have a chicken and an egg, right? And then terraform manage Kubernetes Kubernetes manages through GitHub and others set up company You know what I like the most when you have your maybe self-hosted runners How do you build the image of your self-hosted runners? So I use cloud runners you know, I mean So if I let's say I have a custom custom runner that I need maybe I'm building an Android application I need Android and SDK Android and decay all of those Java and Android tools So I created a Docker image of that But how do I create a Docker image of a runner that is in my cluster? So as you said a chicken and the egg So this is when I like when we say a cloud is just somebody else's computer So then I'm using the cloud runners to build my self-hosted runners images Right? It reminds me a loop. It reminds me I had the CI server Back like six or seven years ago That used to get pretty often it used to get updates that update the infrastructure that it sits on So it would deploy a new infrastructure that would replace itself Right and if something went wrong with that deployment your I mean, there's no system whatsoever The company is gone, but it used to do it and I like I know how dangerous it is But I like it so much because it's everything is completely autonomous. I felt like it's AI Wow, yeah, okay. I want to steer away from the Kubernetes Well, they steer away from the Kubernetes components I just want to ask you an nasty question. All right. We we're also gonna finish soon Not so soon, but soon. I have an nasty question when you started learning Kubernetes, you know 20 years ago When you started learning 40 years ago. Yeah, no for real you have experience with Kubernetes for how long maybe six years seven years Even 10 that something. Yeah ever since I think it was in beta. I don't know. So When you first don't go by anything maybe it's different because I joined To the you know to the whole DevOps world maybe four years ago So things were different back then when I started and when you started when I started DevOps was way more mature Than you started when you did it. It wasn't even DevOps You were a CISOPS right something like that or was it DevOps? No, it was it was DevOps But it was still it was still a baby. It wasn't that mature as now. Yeah. Okay Did you actually when you wanted to start a Kubernetes cluster and run a container in a Kubernetes cluster any in the cloud and whatever? Did you actually go through all the Kubernetes tutorials and read through all the Kubernetes components or did you just skip it? I'm just wondering how did you learn to get into Kubernetes like did you learn the component the control plane how it works or You just started playing with it uh It's it's a lot there's not one answer. I mean Kubernetes has A ton of areas, right? I heard I heard the name. There is one answer. It's the one It's you've done like this is how you learned that's what I want to say every part of it When I learned it I learned it for a purpose Maybe the beginning was actually deploying a cluster the other one. I once taught a course around Kubernetes So I had to learn how Q by DM works and how to start a node and connect it like physically connected To the must to the control plane that you've set up Maybe because we were teaching a course on how to do it on permit, etc So every time it was a purpose so sometimes I went and read the docs When we started I think the most developed tool and was terrible was cops Kubernetes operations. I don't know how much is it being used today So the tooling was a little bit different and I kind of read along the way and then slowly I had to obviously add them in a cluster and then I had to choose whether when Kubernetes is just something else today I'm building an operator So that's why I'm so much into the scheduler and controllers and CRDs, etc But before okay, so before let's say okay, I just want to take you before you started writing your own Maybe admission controller or whatever you're writing. All right before you had to customize something in your Kubernetes cluster Let's take the point only where you wanted to deploy an application and database or whatever Did you go through the Kubernetes components or did you just as I said skip? I'm very I'm very bad with it I do two things when I start working with the system. I Google the I Google I go to the first option and I don't even read through I just look for the for the code snippets You know, you know that you're looking for literally looking for gray areas within the page looking for code snippets and copy-pasting to see what happens That's how I start But but I do have a good answer. I was a good boy after a few days of trial and error that didn't go all that well I decide okay, I don't know Kubernetes. It's not working. I read the book on how to get started with Kubernetes Kubernetes up and running I think and I wrote a blog post and it blew up I got tons of likes and it's still on medium. I live it. I live it in the Show notes. It was I think 2017 And it was great because it wasn't that it's not a genius blog post It's just going through the different components not the components we talked about today. What is a pod? What is a note? Uh, what is a stateful set? What is it? So we'll say the Kubernetes objects Yeah, they're very basic objects if you don't know anything. That's what you learn to begin with So I went through them. I described them And I tried to explain in my own words what each of them does and how I understand it And I figured it was a time it still is but when you write about something and you publish it You want to go through like you want to go seriously through documentation, right? You just don't want to create a bad name for yourself by just making up shit I'm just thinking okay because after hearing our talk today here now That I can I think I could live my whole life without knowing what's a cube scheduler and I'd be fine with it I think Don't have to stop you. I said the exact same sentence yesterday to a marketing team Exactly like word by word. I think I literally said someone can go through their whole life Being a great DevOps engineer admin in Kubernetes and he's totally fine without ever knowing what the controller is So yeah Yeah, because if you don't have You don't have a spare so so my microphone my bug microphone in your office works But you don't really have to to learn I think because what comes into my mind that when people start to get into DevOps or maybe people that are in DevOps or trying to understand What's Kubernetes? So they go through all the components and then it's very intimidating because You're like oh man, there are so many what's cubelet and and people probably already forgot that cubelet is the agent on the Note that but the way I love to remember stuff is Just maybe if you try to build or maybe spawn or create Locally a Kubernetes cluster maybe with mini cube or something like that and then you also mentioned cube ADM Yes, the Kubernetes administrator where you can it's a CLI, right where you can just see a lie that lets you start up Yes, so it communicates with the cube API server, right right right And with that you basically also instantiate it, but yes Yeah, so with that you you actually register worker notes so you can run your workloads on note Walking also yeah, but imagine those things EKS and AKS and GK nobody even tries to do that right and and Now that I think of it. I'm thinking maybe if I was wrong or not okay because initially when I started learning Kubernetes I was like Let's just start running like you. Let's take the code snippets. I knew what is a node what is You know service deployment English English controller. I deployed everything on EKS on Amazon's Kubernetes service I also did it with mini cube locally and I've never Dived in you know, don't I've never dive in. I've I've no idea dove dive even now. Anyways Never went through those Kubernetes components So I'd say to sum up this talk that I think the Kubernetes components are relevant only if you have terrible terrible issues In your cluster and you can't really understand why maybe one pod cannot find another pod and also we skip the service OML we didn't talk about how to communicate between services with DNS So can you mention which which which component does that? Yeah, but I think that's a very customizable component because if you run EKS you get the AWS DNS component if you run on different providers you get that component It's mostly called DNS isn't it? I think it was QPNS and then it was switched with called DNS if I'm not mistaken Right, and you have a bunch of I don't know different ones But I think the documentation lists like four or five very known ones that you can use Okay, I used to work with them. I forgot So there's even another component that we've missed and I could have lived with that, you know Which which is a great way to something I wanted to tell you something very comforting by the way Yeah, because I don't think I know Kubernetes very well to this day every time I learn another thing I just understand how much I don't know at all I didn't do and I heard I told you earlier. I started I listened to an interview with one of the core members It was on the there's literally a Kubernetes podcast. I don't know it's not that good But one of the episodes was amazing And he was speaking about obviously the first days and then how they grew they grew the team how the sigs work The interest group etc, but it is said once and he said nobody really knows how Kubernetes works Nobody he says even me as a core member one of the the founding fathers like one of the first five I think members of the team. He doesn't know how things work simply because it's different groups developing totally different systems I mean the pod controllers and the storage layers and and the DNS stuff It's all completely different and it's completely touched the amazing things that it's it's all playing together And it's customizable and it's still at the hairs do the same conventions and the same rules and everything It's just it's literally magic But nobody really knows how it works. So when you learn something sure the basics fine Yeah, go through them. You'll you'll pick them up do it slowly It's a ton of things to learn But I think just don't get frustrated. I got very frustrated just like you said because it's so much. It's overwhelming So just take it slowly, I guess so bit by bit I'd say if you want to learn Kubernetes just learn what you I think already said it like in previous episode Just learn what you need to learn exactly That's no more than that if you learn more than that people can tell you by the way Not everyone will agree with me people will tell you But this is how you miss stuff because if you learn the more Then you know more so you can do more and I'm like no, no, no, no If I need more I'll go in search and Google for more or Check it for more if it's a thing But I'll do that But I'm not gonna learn Everything and then I'm gonna forget it like in two months because I'm not dealing with it So if it's something that I'm not practically doing I'm not gonna deep dive in it just for fun Maybe I will for self, you know for open source projects, but again, it's practical examples For everything is practical. You will never know everything. I mean it took me like We mentioned DNS like three minutes ago And only now I remember the name that was on the top of my tongue and I couldn't which is calico It's just another DNS plug in Calico flannel. I don't know so many right stuff that I worked for four or five years ago And you just don't remember I don't I don't deal with it. I don't have a reason to deal with it Service mesh remember so don't even go their service mesh and the way to run it through the kernel with Oh, what that was So like it's like it's like you know so many Yes, yeah so many services so many components So I think it's best to learn only the ones you need I also want to give another tip. I think when I learn Kubernetes usually if you do it in the cloud I love seeing what Kubernetes does in the AWS console You know, I also like it when I use the infrastructures code So every time I do something in Kubernetes. I want to go to the I like to go to the Kubernetes like database console because I mentioned earlier if you deploy The application or browser controller or maybe something that controls ingress to your cluster, you know Incoming traffic. It will create some resource in your cloud provider for example AWS So I like seeing that with my own eyes like okay in Kubernetes It is written like this in YAML And how does it look like visually in AWS if I've done it manually You know, I also do that for infrastructures code. Do you also do this thing? I do it and I have something just to add on top of that because it's a great example This week we wanted I mean just to say about you said I want to see what Kubernetes does Within the country because I want to understand what's going on We needed to understand what's what's the scheduler doing? Why is it making different decisions? When is it making these different decisions? Because if it's not if nothing is getting scheduled and you don't actually know what's happening Sometimes you have no way of understanding why things are the way they are within Kubernetes We want them to understand the scheduler. The beautiful thing is when you run within a cluster You can see everything so most likely the components you're looking for are within the cube system namespace So we can just read their logs now. There's a great tool I think we talked about it a ton of times called stern stern is just a great way to read logs In it provides colorful output and it catches a reggae. So I can just run stern and just Substring that's included in many pods and it will just output all the pods that it catches with different colors So it's really easy to follow. So something we did with this week is we ran stern EBS CSI dash CSI because all of the pods that are relevant to the CSI there are tons of them Touch or snapshot their Profisioner lots of them. You learn to love this week. Oh, so we just ran Stern EBS CSI and with the namespace cube system and then you get an output of everything that's going on And it's very easy to understand you can read their perfect logs coming in from each component You can understand why things are the way they are so another tip for following Kubernetes Obviously, that's just the CSI can do it with everything not only applications also internal system You can also read cubelet logs, which is incredibly useful for that you will need to access the logs and then run the journal Journal D something. Yeah, you will need to read the logs on top of the node in order to see what's cubelet doing But it's very interesting. It's very interesting It's an agent running on top of the node and making decisions for you So if you're interested or maybe you have a reason to go there journal CTL Okay, yeah, maybe I may maybe I meant for system D something with the you know demon Like it's always with a D. Yeah. Yeah. Oh merdy. Oh merdeman. You know everything is very deep Okay, so all meld we're about to finish so now we need to move to the corner or you let it move to the corner I think I am I think I am Sounds like a journey. Okay, so let's move to the corner effects Ba-boom ba-boom ba-boom corner of the week. All right, so on a little Which experience did you experience this week or learned or done or saw? I'll do something funny this week. Well, maybe not so funny. You'll decide may do you know Twitter Yes, I do really okay So there's a system called chirper and chirper is Twitter for AI the only users within the system as our AI bots And and you can see okay now I lost my own video. I don't know what happened, but anyway It says internal temperature high you need to cool your that makes it okay Okay, anyway. No, we lost all mail. Yeah. I'm okay. You can still hear me. Yeah, you can call me an internal temperature high Maybe it's me. Anyway, chirper go check it out. It's a cyst. It's a social network only for AI and it's crazy It's also it's funny, but it's scary to see what's going on So that's that's my link for this week Okay, so first I say switch your camera to your laptop camera before we move on to mine and then we'll finish So this will have your face in the end of the video. I will try. I will try you keep on and I will try Okay, so my experience this week was that I like our researchers are collecting data, right and We collect data from history buckets and then we copy it to me like a network drive and whatever So everything is very slow because we are talking about a huge amounts of data and Then I told the researchers listen have an idea Maybe It's gonna be sound like a 90s idea, right? Maybe let's move it to the cloud. Okay, so maybe I will create an easy to instance like a virtual instance in the cloud in In the same region where the s3 buckets that holds all our videos and recordings and data that we collect so Same easy to same machine is next to the s3 And they connect with the VPC endpoint. We've already discussed what's the VPC endpoint in You know previous sessions where you hold all the Network of AWS inside AWS So the issue doesn't go outside to the internet. It just goes in the same network of AWS with the VPC endpoint So Everything was collected from the s3 to the EC2 then we start to start running some scripts and applications in this EC2 And if you knew like the if you compare the speeds so for example if you want to download Huge files from s3 to an EC2 machine. It's like 100 megabytes per second and we used um c6i x color. Okay. I'm not I'm not saying it just for fun Yeah, it's because it's important to know which instance you're using because of the bandwidth, you know So if I use a bigger instance, it's gonna be probably with a higher bandwidth So this instance provides 100 megabytes per second so everything was super fast super fun So if you're collecting data and everything is in s3 and you're working locally Maybe it's time for you to consider to work in the cloud and have a permanent EC2 instance in the cloud. Maybe you can purchase the result instance in the cloud Um, that's it. That's on my end. Oh, all right. We see you Yeah, I did everything I could and now I'm back It's like it's like during when I talked, you know how I felt it's like you are in the sewers Trying to plug Cooling stuff and then I took a shower and I changed clothes and I went back Yeah, okay, so I'm done in your back. Okay, so you're back just in time because we're saying goodbye. Okay and Collect stuff and yeah, okay, so see you next week Yeah, oh, by the way, there's something I want to mention. We forget every time. We have a Facebook page group whatever it is It's a page. We try to put links there. So that's a good reminder for us to put the links there But if you have a page it's a page if you have questions things you want to know place you want I mean, it's a good place to discuss things with us to ask questions So just go there. You can like the page you can send messages. So know that it exists. That's all Yeah, it's a page. It's a page All right, thank you everyone. Yeah, I'll see you next week. See you next week. Bye. Bye. Bye. Bye