This is soo helpful!!! I've been struggling for a while now trying to understand operators and how they work... Thank you so so much! 😊😊
@whyismychannelhandletaken
4 жыл бұрын
Holy smokes, I have been struggling with grasping the stable/prometheus-operator helm deployment. 40K lines of yaml for us. This made it all clear, thank you!
@MarcelDempers
4 жыл бұрын
I would stay clear of stable helm upstream charts. They're great to study implementation details, however they are super over-engineered, hide things they install on your cluster and also make it nearly impossible to upgrade components later down the track. Prometheus eco-system changes too often and doing diff's between charts a few months apart is a nightmare. My DevOps top tip = "Keep it Simple" :)
@cokegen
3 жыл бұрын
@@MarcelDempers true true ... been there ...
@pedronascimento3094
4 жыл бұрын
I love your videos because a learn to use tools with a real-world approach. Thanks to sharing you Knology.
@viktorgadachek3933
4 жыл бұрын
Great work man! Just keep releasing new videos and your channel will be getting more popular. Marry Christmas and good luck!
@tarunprakash2590
4 жыл бұрын
@Marcel If i may request - i am currently evaluating `Integrating Thanos with prometheus operator" to solve known problems associated with prometheus. > Global Query > Longer Retention > HA If you could explore that option and extend this prometheus operator series. Thank you :)
@exit-zero
4 жыл бұрын
Merry Christmas Marcel, nice video, such Prometheus helps DevOps have a better holiday, lol.
@amanmahajan4538
2 жыл бұрын
Thanks for making these videos. Very helpful! :)
@tadeubernacchi3360
Жыл бұрын
Nice video man - What do you mean on 9:35: "Now you can go ahead and create as many prometheus instances as we wanted" - "This is completely up to you how you want to design this" - Did you install the Operator right? Now I need to install a new prometheus deployment in every namespace where I want to monitor? Is that is? How can I install the Prometheus in the k8s cluster where I can monitoring everything that's running in there? Thanks again!
@MarcelDempers
Жыл бұрын
My How to monitor Kubernetes in 2022 video covers this 👍🏼💪🏽 Yes, you use the operator.
@abdulali7692
4 жыл бұрын
Excellent explanation....was struggling a alot now cleared all my doubts....pls make a video for the same using helm chart pls make it soon
@vikashkaushik1557
Жыл бұрын
Very helpful, thanks!
@gadeichhorn
2 жыл бұрын
very clearly explianed, thank you!
@Gandolfof
3 жыл бұрын
Thanks a lot! Your videos are amazing. Liked all of them. I haven't found any good video about RBAC. Please make one!
@Misterad13
3 жыл бұрын
Thx for the explanations ! Congratz !
@varunjain3870
4 жыл бұрын
Great content. Keep it up! Thanks!
@oussamamechlaoui174
3 жыл бұрын
Very interesting content, thanks for this lecture! 5* feedback.
@skchang2239
2 жыл бұрын
really useful video, helps a lot! Thanks you!
@duhhhbruhh3042
4 жыл бұрын
legendary video
@bryanmonsalvatge1696
4 жыл бұрын
How do you recommend exposing this service so that it can be added as a data source to Grafana in a different k8s cluster?
@MarcelDempers
4 жыл бұрын
There are a number of ways you can achieve this. a) Expose Prometheus using an ingress over TLS and something like basic authentication with IP restriction can help. b) Use Prometheus federation to move data to a local Prometheus grafana can connect to. Federate endpoints can also be protected by an ingress controller. c) Another method ive come to learn about is Remote write. Prometheus instances can write data out to an external endpoint in your grafana cluster. This may help reduce number of exposed endpoints you have to protect. Hope this helps 💪🏽
@rajeevghosh2000
3 жыл бұрын
Thanks for the great video. can i ask a quick question. Does prometheus pulls the metrics directly from the containers/pods ? it doesnot need cadvisor ? Also, if I understood correctly, prometheus pulls the metrics using http . Does it mean every container should listen to HTTP requests from prometheus ?
@MarcelDempers
3 жыл бұрын
For container level performance metrics, they are exposed by the kubelet's cAdvisor endpoint and scraped by Prometheus. Similar metrics come from the kube-state-metrics server as well. You only need to expose HTTP endpoint on your service if you have custom application metrics that you want to grab and then you'd use a ServiceMonitor to scrape those
@Fidellio369
24 күн бұрын
Can you speak of the Thanos sidecar and other Thanos services offered with the Prometheus operator?
@DiscoDaveDance
4 жыл бұрын
Great video. But wearing an Arch Linux T-shirt whilst recording on Windows? :D
@korbkrys5868
3 жыл бұрын
Can you do a video on the crunchy Postgres operator or Zalando operator?
@zaheerhussain5311
3 жыл бұрын
Hi Nice explanation. Do you share deployment of prometheus operator using dynamic provision storage. Regards Zaheer
@jayeshthamke7860
4 жыл бұрын
Great material! How to aggregate the metrics if we have multiple Prometheus instances running into cluster? And can we aggregate metrics from k8s multi-cluster may be deployed in Gardner? Thanks in Advance.
@MarcelDempers
4 жыл бұрын
Thank you for the kind words 🤓 There are a few approaches you could take. If you are running multiple prometheus instances you can either replicate the data to a central prometheus instance by using the /federate API. Prometheus instances can scrape each other to move data out to a central one for dashboarding. Another approach is have every prometheus instance push its metrics out to a central storage using the "remote-write" feature. You can then use that data store as a data source for dashboarding. Hope that helps 💪🏽
@tarunprakash2590
4 жыл бұрын
Thanos querier component is exactly designed to do - merge and deduplicate similar metrics. Visit thanos.io to know more
@bronsonvandrades
3 жыл бұрын
Is there any document for Prometheus to start from scratch? i mean i have knowledge on Kubernetes now since i have CKA exam completed. I need to know how to install Prometheus on nodes exactly do we need to install on master node or worker node? or we need separate node to get Prometheus installed? from this videos i understood that Prometheus is a tool which i will pull the metrics logs from the kubernetes objects and then we need some visualization tools like Graphana to display the logs pulled by Prometheus do we need lots of storage space to get these logs stored ?
@DrBustenHalter
4 жыл бұрын
Like your style! If we have multiple replicas of prometheus in a deployment, for resilience.. they will double-up the same stats and network iops? Do you recommend only 1 prom pod per deployent / stateful set? Have you tried the stable/prom-operator helm that bundles grafana, kube-state-metrics, etc..? it looks good but seems to run 1 prometheus that pulls in everything in that infrastructure scope. I guess you extend it per your example here .. and add that new prom service DB / datasource to grafana? A lot is not transparent in the helm deployments... quick but you lose control.
@MarcelDempers
4 жыл бұрын
One thing to note is that Prometheus architecture has high availability in mind. You dont need many instances scraping the same targets. What I would recommend is perhaps slicing it up to where you have 1 instance per subset of targets. Like 1 that looks after node-exporter. Perhaps 1 that is dedicated to kube-state-metrics, 1 for API server, something along those lines. By distributing the responsibilities of Prometheus, you get improved resilience. Same goes for 1 prometheus per group of micro services. Another thing to note is that if prometheus is down, you persist data on something like a persistent volume (statefulset). You won't lose metrics. As long as Prometheus comes up in time it will rescrape and continue to collect stats that may have been missed during downtime. You can also increase availability by federating all the instances data across to other Prometheus instances with the /federate endpoint. Dashboards can point to a federated copy too, increasing resilience even more. Prometheus is complex, but extremely flexible and the above statements make it very resilient, when setup with availability in mind.
@DrBustenHalter
4 жыл бұрын
@@MarcelDempers Thanks for the reply!! I agree. I was working with the helm stable/prometheus-operator which comes with a load of useful stuff to get you up and running (Alertmanager, Grafana with Dashboards, all the K8 infrastructure ServiceMonitors, etc). I was worried it wouldn't be modifiable - but you can add a new Prom instances the way you show here for the application layer etc. You can even move off kube-state-metrics, etc to dedicated promethei too if that 1 instance gets too big. Persistence works fine and new prom's all get added as stateful sets. I just create a new grafana data source as they get added. So it's just a helm release with a few patches applied to customise it... very cool!! Looks like I've done a month of work 😆👍
@m19mesoto
3 жыл бұрын
Hello, I think, now is the time to do Thanos-Operator.. or just thanos.
@robl39
11 ай бұрын
I wonder how much Marcel can bench press
@OfficeNothanks
3 жыл бұрын
Can you use the Prometheus operator to monitor for example barebones machines running node exporter on them?
@brijchavda
3 жыл бұрын
Would like to see thanos on kubernetes.
@SteelerLemon0
4 жыл бұрын
How do you add Kube state metrics to Prometheus with this set up?
@MarcelDempers
4 жыл бұрын
video coming soon 👍
@satishkad1804
2 жыл бұрын
Do you have anything on getting spark application metrics using prometheus?
@Punjab_the_land_of_five_rivers
4 жыл бұрын
Hello sir , i got stuck in helm chart of prometheus-operator. In my project i deployed my php application under kube-system namespace. And install the prometheus operator helm chart under the monitoring namespace. Now i want to monitor my application (not the nodes and kube-system just the applications). What will be the direction path for it?
@vtvvnha2230
3 жыл бұрын
how to run 1 prometheus instance to monitor 2 seperate services in 2 seperate namespace
@MarcelDempers
3 жыл бұрын
Take a look at ServiceMonitors , you can select a service amespace to scrape and tell Prometheus which ServiceMonitors to select
@georgelza
3 жыл бұрын
executed the operator directory... had errors, did a tear down, is there a order in which i can execute the yaml files manually, after having created the namespace.
@sol5179
4 жыл бұрын
I think background musics between chapters are little bit loud? Almost died of heart attack.
@georgelza
3 жыл бұрын
... the prometheus-operator directory does not seem to be part of your directory structure anymore ?
@MarcelDempers
3 жыл бұрын
All k8s prometheus related items are under👉🏽 ./monitoring/prometheus/kubernetes/README.md
@ankitagarwal6014
4 жыл бұрын
i am a devops fresher and i have just come across prometheus and grafana. is there any step by step learning video on these technologies?
@MarcelDempers
4 жыл бұрын
Nice! Welcome, here is a playlist with step by step content and source code kzitem.info/door/PLHq1uqvAteVuEXCrRkPFWLXRKWNLOVUHn 💪🏽
@ankitagarwal6014
4 жыл бұрын
@@MarcelDempers thanks 😊..also can you make some videos on managed services offered by cloud providers like EKS and AKS?
@iposipos9342
2 жыл бұрын
Hi, thanks for the nice video. i have a question. if after setting up Prometheus for my cluster, i later on added decided to scale my cluster by adding new VMs, will the existing node exporter pick the metrics for the new VMs? or will i have to configure the node exporter for the new VMs? Thank you
@MarcelDempers
2 жыл бұрын
node-exporter should be running as a daemonset, so it will scale to the new machines when they start up
@iposipos9342
2 жыл бұрын
@@MarcelDempers okay, thank you
@supratimmukherjee8519
3 жыл бұрын
For grafana, the pods do not come up and I see the error below for the pod events Normal Scheduled 2m29s default-scheduler Successfully assigned monitoring/grafana-f667ddbdc-2pk2p to ip-10-2-100-235.ec2.internal Warning FailedMount 26s kubelet, ip-10-2-100-235.ec2.internal Unable to attach or mount volumes: unmounted volumes=[grafana-dashboard-nodeexporter], unattached volumes=[grafana-dashboard-node-cluster-rsrc-use grafana-dashboards grafana-dashboard-k8s-resources-workloads-namespace grafana-dashboard-nodeexporter grafana-dashboard-proxy grafana-dashboard-namespace-by-pod grafana-dashboard-namespace-by-workload grafana-dashboard-node-rsrc-use grafana-dashboard-statefulset grafana-dashboard-k8s-resources-node grafana-datasources grafana-dashboard-scheduler grafana-dashboard-k8s-resources-workload grafana-dashboard-k8s-resources-pod grafana-dashboard-pods grafana-dashboard-workload-total grafana-dashboard-nodes grafana-dashboard-prometheus-remote-write grafana-storage grafana-dashboard-cluster-total grafana-dashboard-controller-manager grafana-dashboard-kubelet grafana-dashboard-prometheus grafana-dashboard-apiserver grafana-dashboard-k8s-resources-namespace grafana-dashboard-pod-total grafana-dashboard-persistentvolumesusage grafana-dashboard-k8s-resources-cluster grafana-token-xs4fs]: timed out waiting for the condition What am I missing here? I am simply applying everything from your repo.
@MarcelDempers
3 жыл бұрын
This does seem strange. Grafana mounts a ton of configmaps where the dashboard definitions are stored. Make sure the configmaps are created. You have to kubectl create not apply on the configmaps because the files are very large. Also make sure you're creating configmaps and the deployments in the right namespace as the tutorial suggests. Check kzitem.info/news/bejne/wpink2p8e6BhmaQ too for more info
@supratimmukherjee8519
3 жыл бұрын
@@MarcelDempers Yes thanks, using create instead of apply works but what is the rationale behind that, you mentioned to use create instead of apply for large config map files, I am wondering if I were to run this in a CI pipeline like argo, it will do an apply as part of sync and will fail, any thoughts on this?
@68naveeng
3 жыл бұрын
Hi Could you plz guide us to install snmp exporter in kubernetes
@andrewfigaroa7031
2 жыл бұрын
This works for me to get things installed, but how do I add my targets (my custom Apps)? I know with prometheus VM I populate my static_configs in /etc/prometheus/prometheus/prometheus.yml file. but I have no clue how to accomplish this with the helm approach. Also, I have not been able to locate any documentation on how to do this.
@MarcelDempers
2 жыл бұрын
Got two videos coming soon that address this. You want to learn about Service monitors
@andrewfigaroa7031
2 жыл бұрын
@@MarcelDempers awesome! Thanks
@gouterelo
4 жыл бұрын
Hello Marcel, great videos in your channel and great you have a series of prometheus and grafanna !!! One question, i try to apply your yamls and i get this error... error: unable to recognize "service-monitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1" I guess it will be the version of my k8s cluster (18.0)... !! Greetings from argentina :D ! I deploy it again after all deploy and it deploy it well... depends of the cluster speed to create all the instances :D !
@MarcelDempers
4 жыл бұрын
Thanks mate💪🏽 Just a notice, the prometheus operator needs to be running first. When it runs, it creates the servicemonitor CRDs so that you can apply the service monitor. You can only apply the service monitors once the operator has initialized properly 🤓If you apply all the YAML files too fast, the ServiceMonitor CRD will not have been created and you will face that error. hope that helps
@gouterelo
4 жыл бұрын
@@MarcelDempers Thanks for the advice Marcel ! another question... i have problems to scrape kubelet if i deploy all your yamls... and i cant found where the error is, i think because your yamls are for minikube ? i deploy a k8s HA cluster on prem... cheers !!
@chasim1982
4 жыл бұрын
Really Nice explanation, I am getting this error ---> error: unable to recognize "service-monitor.yaml": no matches for kind "servicemonitor.monitoring.coreos.com" in version "monitoring.coreos.com/v1"
@MarcelDempers
4 жыл бұрын
Thank you 💪🏽 Older version of the operator will create the CRDs for service monitors automatically when starting up. Once the operator is running you can retry adding service monitors. OR - You can deploy the CRD's separately like I do in this video kzitem.info/news/bejne/wpink2p8e6BhmaQ
@prasantcr
3 жыл бұрын
I have a question, can we edit the prometheus.yaml file, I mean the configuration file. not the one that is in the repository. If yes how can we do that..?
@MarcelDempers
3 жыл бұрын
The Prometheus CRD provides the interface to configure prometheus so you dont have to touch and manage the underlying configuration. ServiceMonitors are also an interface to this. You can check the docs here github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/api.md#prometheusspec
@prasantcr
3 жыл бұрын
@@MarcelDempers So how do I edit the present configuration of the current scrape configuration. ?
@prashantpathak3175
3 жыл бұрын
Can I use Prometheus without installing or deploying Prometheus Operator?
@MarcelDempers
3 жыл бұрын
Sure you can. You can run it as a deployment and simply keep data in memory or If the data is important to keep you can use a statefulset and appropriate volume to persist data to.
@prashantpathak3175
3 жыл бұрын
@@MarcelDempers Can you please make a video where we monitor the ingress controller for nginx and then get the required matrix. In one of the video you have made is only to use ingress controller and there I did not understand how to get the domain name?
@SteelerLemon0
4 жыл бұрын
Does the python application have to be exposed to Prometheus metrics for this to work?
@MarcelDempers
4 жыл бұрын
An application needs to expose a "/metrics" endpoint to a Prometheus instance (not the operator) for this to work. A service monitor needs to select the application service correctly too
@SteelerLemon0
4 жыл бұрын
Marcel Dempers -DevOpsGuy thank you for the quick response!
@albuch
3 жыл бұрын
Your source code links do not work. On this page there is a lot of broken links associated with you series ( github.com/marcel-dempers/docker-development-youtube-series) click on almost any Source link. (Found it)
Пікірлер: 78