🔴 - To support my channel, I’d like to offer Mentorship/On-the-Job Support/Consulting - me@antonputra.com
@vishnuvardhanvandavasi9998
10 ай бұрын
when it will come please let us know @AntonPutra
@Fidellio369
24 күн бұрын
Hey Anton, Is your mentorship still available? (And yes, I'd be happy to pay :) )
@cokegen
Жыл бұрын
Producing these had to take quite some time ... thanks for doing them, learning a ton from these prometheus videos you recently posted 10/10
@AntonPutra
Жыл бұрын
Thank you, I'm glad that you find it useful!
@madbbb
Жыл бұрын
Spasibo Anton! I was googling how to secure remote write on Thanos Remote Write/Prometheus Agent and you just released this video 3 days ago.
@AntonPutra
Жыл бұрын
pojalusta, source code in the description
@kannan7709
6 сағат бұрын
Amazing video and lecture..keep going .
@AntonPutra
6 сағат бұрын
thank you!!
@mitesh6872
8 ай бұрын
Perfect compilation of each use case and detailed explanation. I haven't seen such a video on Prometheus and Thanos before. Thank you very much Really Helpful 👍
@Leonid.Shamis
Жыл бұрын
Hi Anton, thank you for the informative and educational content! I look forward to your next video about preparing a Production ready Thanos setup using Amazon EKS Cluster remote write endpoint and deploying stateless Prometheus in Agent mode and moving local alerts to a Centralized Thanos.
@AntonPutra
Жыл бұрын
Thank you, will do
@675FresH
Жыл бұрын
Thanks Anton! Helpful as always!
@hachastico674
4 ай бұрын
This is super helpful, thank you a LOT
@AntonPutra
4 ай бұрын
welcome!
@denisrazumnyi6456
Жыл бұрын
Well done !!! Thank you for your job !
@AntonPutra
Жыл бұрын
Thanks =)
@PattamaW
Жыл бұрын
super informative and useful tutorial, I can't wait for the production setup!
@AntonPutra
Жыл бұрын
Thanks, well it's very similar but implementations would vary depending on the cloud you use.
@bender5326
Жыл бұрын
As usual another great tutorial! N.B. I hope you will present a video about VictoriaMetrics) VM more productive and has great architecture
@AntonPutra
Жыл бұрын
Thanks, will do!
@minimalniemand
Жыл бұрын
great tutorial, thanks!
@AntonPutra
Жыл бұрын
Thank you!
@saadullahkhanwarsi5853
Жыл бұрын
Thanks anton for all efforts, But we guys still waiting for remote Prometheus setup/cluster setup monitored through Grfana via Thanos. It would be great help Again thanks 🎉❤
@AntonPutra
Жыл бұрын
Thanks will do eventually =)
@emergirie
Жыл бұрын
You got us used to doing everything in terraform was you tired😂 it’s a joke thanks for the sharing
@AntonPutra
Жыл бұрын
Better doing it by hands for learning:)
@sundeepgarg3502
Жыл бұрын
you are genius
@AntonPutra
Жыл бұрын
haha
@monitorcamera8850
Жыл бұрын
awesome video
@AntonPutra
Жыл бұрын
Thanks, took me a while to record =)
@itspngu
Жыл бұрын
we'll use an S3 bucket to store prometheus magics. :)
@smitzaveri12
6 ай бұрын
Hi @Anton .. why do we have 2 receivers ? Is it because to distribute load? also can you clarify this : 1. Lets say we have more than 2 prometheus instances and we have 1 thanos, should we use multiple receivers i.e more than 3 or 4? 2. Is it wise to assign 1 receiver to 1 prometheus meaning receiver 1-> prometheus 1, receiver 2-> prometheus 2 etc?
@clementbenedetti724
Жыл бұрын
Hi, thank you for the video, very clear as always, cannot wait to see the next video for Thanos in production, I hope soon. Are you gonna use the prometheus agent in the next video?
@AntonPutra
Жыл бұрын
Not sure when I'll create a following video cause I don't see a lot of interest in it here. I have one video on prometheus agent (kzitem.info/news/bejne/t6-P2IiegoB0nZw), set up remote write is the same as with other prometheus.
@ukaszl.9943
Жыл бұрын
@@AntonPutraAmazing job! I wait for Thanos in production. I try to find some tutorial to do it but with no luck. I'm a new one with Kubernetes and Operators world and I don't know where to change prometheus configurtation to create this agent, write-remote solution. Please help :)
@nitishchauhan7377
Жыл бұрын
Hello Anton Good Video , a query when are you planning to release Thanos Production Setup ?
@AntonPutra
Жыл бұрын
I'm thinking about it since lot of people asked. Probably soon, but all the pieces for that in this video. Just need to use public load balancers pretty much and of course configure IAM for S3
@nitishchauhan7377
Жыл бұрын
@@AntonPutra Oh that's great. Also I had a query, using remote write I'm storing data in thanos , now on the grafana level what changes should be done so that the tenant-A are able to see metrics of clusters running in tenant-A and tenant-B are able to see metrics of clusters running in tenant-B.
@AntonPutra
Жыл бұрын
@@nitishchauhan7377 you can use tenant label on that metric
@denisrazumnyi6456
Жыл бұрын
Thanks!
@AntonPutra
Жыл бұрын
Thanks for the support @denisrazumnyi6456!
@pier_x0
Жыл бұрын
Hi Anton, great topic. Thank you so much!!! I can see on the cover #1, have you planned any further tutorial on that topic? Do we have any chance for a Federated Prometheus arch tutorial?
@AntonPutra
Жыл бұрын
Thank you. However, I believe that Prometheus federation may no longer be widely used. Is my understanding incorrect?
@pier_x0
Жыл бұрын
@@AntonPutra yeah, I think you're right. Maybe Thanos fix that issue to collect metrics from different Prometheus Is it correct?
@AntonPutra
Жыл бұрын
@@pier_x0 yes it’s a main use case for thanks to give you global view
@starytomas
11 ай бұрын
Hi, when are you planning to make a video about thanos and prometheus in agent mod please? Thank you very much for your answer. The current video was great, I would also like to see more about prometheus agent mod and Thanos.
@AntonPutra
11 ай бұрын
Thanks, it's very similar and I have examples somewhere in my GitHub repo. You just need to provide a couple of additional flags and you get stateless, well almost stateless prometheus
@dhartipatel8769
4 ай бұрын
Super helpful video @AntonPutra . I have replicated this set up for my project. I have deployed on k8s v1.26.5 cluster. The only difference is I am using higher versions of Prometheus (v2.48.1); Prometheus-operator (0.71.2). I also upgraded the CRDs as per the operator. The issue is the I am not able to fetch any active prometheus target. I've checked all the configurations, deployed pods are running up and fine. The servicemonitor has correct labels(same as your src code). Any help appreciated ! Best Regards.
@dhartipatel8769
4 ай бұрын
Forgot to add, I am checking the target at the local Prometheus UI :) There are no error logs as well, neither in operator pod nor in the prometheus-staging pod. I see one expected log missing as below when I compare it with your video: level=info msg="discovery manager scrape" discovery=kubernetes config=serviceMonitor/monitoring/prometheus/0 msg="Using pod service account via in-cluster config"
@user-cw8gt9yk3l
11 ай бұрын
Thanks for your tutorial. I would like to know in real production scenario, are there any ways to expose minIO securely without port forwarding?
@AntonPutra
11 ай бұрын
Sure, Minio is a distributed web server that mostly uses HTTP and websockets. Use an ingress or simply a load balancer to expose Minio.
@nishasati8249
8 ай бұрын
Awesome video. Thanks a lot for sharing this. Could you please share the production ready multicluster monitoring setup in AKS? What will be required components to be setup in centralized cluster and other individual cluster? Any document would be a great help.
great tutorial as always. when is the production setup coming pls?
@AntonPutra
Жыл бұрын
Thanks, it's not much different just use external dns and more shards =)
@rafael.aloizio1769
Жыл бұрын
Hey Anton, I'm really curious to know this, during your videos you usually organizing files with numbers in front of them, the question might be stupid, but I'll ask, do you use this only for the tutorials, or do you have a reason behind this, and follow this in your production coffee as well?
@AntonPutra
Жыл бұрын
Hi Rafael, no just for tutorial for files to be in order. Don't use numbers in your production code, it would look like a beginner's code =)
@rafael.aloizio1769
Жыл бұрын
@@AntonPutra oh ok, thanks for the answer, I'm a beginner though haha 😂 in the DevOps world at least.
@BingedFriends
3 ай бұрын
Hi @AntonPutra, Thanks for the video. Did you get a chance to prepare the Production ready Thanos setup using Amazon EKS Cluster remote write endpoint and deploying stateless Prometheus in Agent mode and moving local alerts to a Centralized Thanos.
@AntonPutra
3 ай бұрын
Not yet, but I'll definitely create one soon
@xXKingofDiamondsXx
2 ай бұрын
@@AntonPutraI would love to see it done in AWS! I’m trying to continue off what I’ve learned from this video. Do I now need to expose the Querier from each cluster to Grafana? I’m so grateful for the content you put out. MANY thanks and cups of coffee for you friend!
@AntonPutra
2 ай бұрын
@@xXKingofDiamondsXx I may refresh it soon, maybe with Thanos, Cortex, or even Grafana Mimir. I haven't decided yet which one to cover first.
@sre-monitoring
8 күн бұрын
Nice tutorial. Whats the recommended way to install thanos on production grade level? Do we have respective helm chart from thanos community?
@AntonPutra
8 күн бұрын
It doesn't really matter how you deploy it for production-whether using plain YAML, Kustomize, or Helm. Just make sure you use the same approach throughout your entire infrastructure.
@njg120
8 ай бұрын
hi , good video. You were talk about a video of how configure prometheus agent , IRSA with bucket s3 and alb aws for exposing remote-write-svc , where are those videos ? thanks
@AntonPutra
8 ай бұрын
I haven't recorded them yet, sorry.
@user-hb4kl2wd8i
5 ай бұрын
Nice video Anton, do you have something for thanos receive distributor and thanos receive configuration. Or the one which is video is the same thing. For me the remotewrite does not work if the distributor is enabled via the helm chart. Any help is highly appreciated.
@umeshgholve35
9 ай бұрын
Hello anton, thanks for video. can you please give steps for aws eks cluster monitoring with prometheus thanos setup.
@AntonPutra
9 ай бұрын
Well, you need to use IRSA to grant the Thanos storage gateway read/write access to the S3 bucket. The rest of it is more or less the same. Are there any specific issues?
@anashamdan5237
Жыл бұрын
Please complete the series about devops interview questions
@AntonPutra
Жыл бұрын
ok
@ignatiussahmofor3385
8 ай бұрын
Nice video . I clone you repo but was didn't see the yaml files you were using.
@AntonPutra
7 ай бұрын
Sorry for the delay. All of the YAML files are here: github.com/antonputra/tutorials/tree/main/lessons/163.
@m18unet
Жыл бұрын
Hi Anton. Thanks a million times for the awesome series. I have a question. Let's destroy the minio and set the "tsdb.retention" value like "180day" in the receivers. Is this bad practice? I don't want to use any object storage(S3, Minio etc.). I am asking this question for the Remote Write approach.
@AntonPutra
Жыл бұрын
No, it's fine and it can eliminate the bottleneck. Just make sure you set up sharding from day one. Also, if you decide to use Thanos Ruler, it's more reliable without S3 and the store gateway.
@m18unet
Жыл бұрын
I completely understand you. Thanks for your detailed explanation Anton 😊
@m18unet
Жыл бұрын
By the way, why might I need to install the thanos-ruler? I don't fully understand this component.
@AntonPutra
Жыл бұрын
@@m18unet For example, if you run stateless Prometheus in 'agent' mode, no data is stored locally. This means you cannot run Alertmanager locally as well. The Ruler is used to run alerts on the global state, which also has some drawbacks. You can read more about this in the official documentation.
@SangeetaMeena-i5q
Ай бұрын
Awesome video! I was trying to follow as in video, I was stuck at few points. For first step, when thanos was disabled and 2 pods of prometheus-staging-0 were created, my expectation was to be able to see targets when prometheus-operated was port forward on port 9090 to my localhost. I didn't see that. Could you please let me know why? Second after adding thanos sidecars without mutual tls, on port forward querier on 9090 on localhost, I did see sidecar in my Stores but I wasn't able to see any Prometheus metrics as you showed . Please guide.
@AntonPutra
Ай бұрын
Thanks, there are a lot of labels that must match in order for the target to show up. I have another video on my channel covering Prometheus Operator, it may help. For the second question, if you don't have any targets, you won't have any metrics in Prometheus.
@vishnuvardhanvandavasi9998
10 ай бұрын
we need a setup between two eks cluster metrics with prometheus thanos ha and stored in s3.... we are awaiting for this log from 4 months
@AntonPutra
10 ай бұрын
Sorry, I didn't get a question.
@gabrielportela6544
Жыл бұрын
Thanks for the tutorial Anton! When the next video will be released?
@AntonPutra
Жыл бұрын
It appears that it's a very niche topic. Do you think there is an interest in continuing and providing prod ready setup?
@gabrielportela6544
Жыл бұрын
@@AntonPutra I think so. I could not find much helpful content about Thanos implementation on KZitem and you are always providing great tutoring. On top of that, the only other tutorial I watched also talked about a second video that was never released
@AntonPutra
Жыл бұрын
@@gabrielportela6544 okay, but I still need to finish my terraform series.
@XRoydX
11 ай бұрын
I'd also love too see the video ❤ Thank you so much for your work
@508mamidianilkumar6
Жыл бұрын
Hi Anton, Could you please do a video on how to use Prometheus in agent mode and forward metrics to Thanos Receive with scaling on both Agent and Receive side?
@AntonPutra
Жыл бұрын
I have one video on prometheus agent (kzitem.info/news/bejne/t6-P2IiegoB0nZw), set up remote write is the same as with other prometheus. Just use a code snippet from this video.
@user-lo1sz6cz8r
Жыл бұрын
Hi Anton, thank you for sharing your videos. It has been incredibly helpful and expanded my knowledge. I followed your tutorial and successfully deployed Thanos with remote write Prometheus. Everything appears to be functioning correctly, except that I am unable to view any targets in the Thanos UI. Could you suggest a reason for this issue? btw, I used helm charts instead of manifests, which are much easier Furthermore, I have another question regarding best practices for the Alert Manager. Should I transfer all alerts to the ruler or should I deploy the Alert Manager with each Prometheus instance? Thank you in advance for your assistance.
@AntonPutra
Жыл бұрын
Thanks! Well, first of all, you should not see any "targets" in the Thanos UI; you should only see "Stores". Targets should be visible in the local Prometheus UI. Second, Thanos recommends using local Prometheus and Alertmanagers because they do not rely on external components for queries to succeed. In my opinion, if Thanos is well monitored and any issues can be immediately identified, then using a global Ruler is acceptable.
@backpack2861
10 ай бұрын
Thanks for this video and sharing your knowledge, I just have a question about it, you mentioned that remote write is your favorite solution and I really like how this is working but I was investigating about remote write and the documentation says that is recommend only in cases of pushing metrics is the only solution. What do you think about it? I don't see any problem using remote write but maybe I'm ignoring something here (I'm newbie in this)
@AntonPutra
10 ай бұрын
I've been running thanos at scale 10+ envs for a couple of years, based on my personal experience remote read almost lags, so I found in order to provide better user experience for developers and other devops team members is better to use remote wrtie
@leeren_
Жыл бұрын
Looking forward to the followup video for production endpoints! Is that hard to setup?
@AntonPutra
Жыл бұрын
No, it's pretty much the same. By 'production', I meant that I was going to show how to use cloud-specific components, such as IRSA, to set up permissions to access an S3 bucket, etc.
@leeren_
Жыл бұрын
@@AntonPutra Got it. To allow multi-cluster Thanos, is it as simple as making the receiver endpoint external? Would exposing it through an ingress and figuring out how to use cert-manager fix this? Any gotchas or resources? Thanks again, this tutorial was AMAZING
@AntonPutra
Жыл бұрын
@@leeren_ yes you need public DNS for each receiver endpoint, but those endpoints use custom protocols and ports so you need dedicated load balancer or you can tcp service on Nginx to share one
@leeren_
Жыл бұрын
@@AntonPutra For AWS do you recommend using ACM or just doing cert-manager ?
@AntonPutra
Жыл бұрын
@@leeren_ Either way, you don't need a public certificate. You need to create a CA, and you can import it to Cert-Manager to automatically renew your private certificates. The same applies with ACM.
@mamitadieng4166
9 ай бұрын
Hi Anton Putra. Thx for thiw video very usefull. If I decide to use the opensfhit cluster prometheus native thanos query as datasource for an external grafana which is in a nerwork outside the openshift one, do I have first to open the flow?
@ahmadbanisalameh9285
9 ай бұрын
Thank you for your videos very insightful , I have a question, I have multiple clusters each cluster has a prometheus deployed to it and I have a monitoring cluster that has a centralized thanos, the prometheuses are in agent mode with enabled remote write, do I need to deploy a receiver for each prometheus ?, using thanos helm chart and for prometheus kube-prometheus-stack, and if so do each receiver need a qeurier or I can connect all of them to the same querier, each one is gonna save to a different bucket does the same needs to happen for the store gateway ?, and if you have any suggestions helping me architect this setup. Thank you again
@AntonPutra
9 ай бұрын
Sure, you can deploy a single receiver (it can be sharded) and point all your prometheus agents there. Also you need at least 1 querier to query that receiver for short term metrics and store gateway in case you use object store for long term storage.
@ahmadbanisalameh9285
9 ай бұрын
but how the receiver is gonna know which bucket to upload to if its a single one and all my four prometheuses write the metrics to it , for example the prometheus on dev should be uploaded to a dev bucket and so on @@AntonPutra
@the-learner-le5y
8 ай бұрын
Hello Anton Putra, When can we expect the video on production ready Thanos setup?
@AntonPutra
8 ай бұрын
I don't know yet; it's very similar. You just need to use IRSA to configure permissions for S3. Do you have any specific use case in mind, such as 1 cloud, 2 clouds, or perhaps 10 environments?
@the-learner-le5y
8 ай бұрын
@@AntonPutra Thanks for the reply, Planning to setup in Multiple AKS clusters where the receivers will be behind ingress. Anyways let me set it up.
@Kavinnathcse
3 ай бұрын
very nice tutorial .. so whats the difference between thanos vs grafana mimir ?
@AntonPutra
3 ай бұрын
thanks. i haven't used it yet, but I'll definitely test it and create a tutorial, maybe a direct comparison soon
@Kavinnathcse
2 ай бұрын
@@AntonPutra Thanks a lot.. one more thing as per official doc looks like thanos remote-write is not recommended for single tenant. if so, do you know the reason behind that?
@AntonPutra
2 ай бұрын
@@Kavinnathcse Prometheus generally does not recommend using push methods except when necessary. Based on my experience managing 20+ environments and almost 100 Kubernetes clusters, remote read is a bit slow since each time you open Grafana, Thanos needs to query the remote environment. To improve user experience and speed up queries, we decided to use remote write.
@nickcarlton4604
Жыл бұрын
Thanks Anton, will you be doing a video on a HA Thanos Receive setup? I can get it working with a single Receive pod but when I setup multiple I start getting internal server errors etc
@AntonPutra
Жыл бұрын
I cover it in this tutorial. here is an exampe - github.com/antonputra/tutorials/blob/main/lessons/163/hashring.yaml 41:25 Configure Thanos Receiver Sharding
@nickcarlton4604
Жыл бұрын
@@AntonPutra Thanks for that, the issue I see is when one of the receivers goes down for whatever reason the metrics it has are lost and Prometheus stops remote writing and throwing 500s because it cannot communicate with a host from the hashring. Does that make sense?
@AntonPutra
Жыл бұрын
@@nickcarlton4604 Yes, but when the receiver recovers, prometheus will write all the missing metrics (if the receiver was down for few hours). You can set the replication if you want - github.com/antonputra/tutorials/blob/main/lessons/163/receiver-1/statefulset.yaml#L53 I'm not sure if HA is posible in this sense. You're right it's a receiver sharding for scalability and has nothing to do with HA (high availability when one replica goes down). I know that HA mode on Prometheus is possible by running multiple independent instances and use external labels for deduplication (it may or may not use different receiver shard...)
@nickcarlton4604
Жыл бұрын
Thanks, I guess it would be more a case of if I had 2 receivers and a hashring, when a receiver goes down, how can I still remote write to the remaining one? Or would it be better to have two receivers without a hashring, let them work independently via a clusterIP service but ship the data off to object storage faster (a few minutes) so then it can be consumed from store gateway?
@AntonPutra
Жыл бұрын
@@nickcarlton4604 Not sure about that, I use 5 shards in prod and don't have any issues. Please try it and let me know if it works
@VinayKumar-ek6bq
5 ай бұрын
Hi, i am doing this on on-prem and i don’t have any storage i can use except PVC, is there any way i can use thanos for this ? If yes then what will be the sidecar config ?
@souvikdas2457
9 ай бұрын
@AntonPutra : From where to get these yaml's??
@AntonPutra
9 ай бұрын
I found them in the official repository a while ago and made a few updates. Are there any concerns?
@AloisioBilck
Жыл бұрын
Hi Anton, why don't you use helm to install prometheus?
@AntonPutra
Жыл бұрын
You can use helm, nothing wrong with it
@salamander-101
Жыл бұрын
i think grafana mimir better than thanos at scale and easy to scaleout
@AntonPutra
Жыл бұрын
I've never used it before but I'll give it a shot
@user-wt8ji9dg3t
Жыл бұрын
Question : Do we have any Helm charts or Operator for the Receiver Stack Installation or CRD's is the only way for now ? I checked the Official Bitnami Chart which supports the sidecar approach but nothing on the Receiver Any ideas ?
@AntonPutra
Жыл бұрын
I don't think so, there is one for just prometheus - github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
@evelayudham3247
11 ай бұрын
Hi Anton, got a question, How is thanos querier connected to prometheus sidecar. What values is to be passed in querier.stores list ?
Hi Anton, How to configure the prometheus to handle 4000 to 7000 Pods cluster? When we put current config and setup remote write, prometheus starts using 45GB+ RAM. can you please tell the best way to optimize it?
@AntonPutra
7 ай бұрын
Two ways: 1. Manual sharding allows you to have multiple Prometheus instances querying different targets. 2. Prometheus has a sharding feature, but it's in beta; you can take a look.
@mitesh6872
7 ай бұрын
@@AntonPutra Thanks. 1. so it will be 2 prometheus instance of 22 GB RAM but usage of 44GB is expected,right? One more thing related to put the datasource in grafana. Do I need to create 2 datasources. 1. Prometheus 2. Querier. Is there a way I get all the data of prometheus,Querier and Storage Gateway with single datasource?
@emergirie
Жыл бұрын
Do you think their is an interest to use thanos when implemented managed Prometheus ?
@AntonPutra
Жыл бұрын
based on my experience self managed thanos up to 100x cheaper. cadvisor and node exporters create lots of metrics with high cardinality (lots of label combinations) can be very expensive with managed services like managed prometheus or any other data dog sfx you name it
@madhankannan8022
10 ай бұрын
Hi , I am not able to find the video for thanos setup in eks
@AntonPutra
10 ай бұрын
It's very similar to this one but I haven't released it yet
@harshitanand6209
Жыл бұрын
Hey Anton, Followed your video and implemented the thanos write method, all the pods are running without error, but I am still not able to see any metrics in thanos querier :( Can you point me what I could be missing here ?
@AntonPutra
Жыл бұрын
I would suggest starting with the minimal setup and trying to inspect every component for any errors.
@phyohtetpaing44
Жыл бұрын
Hi sir can you make a database benchmarking (mongo db vs mysql vs postgresql)
@AntonPutra
Жыл бұрын
Sure, I was working on one between MySQL and Postgres already
@rahulchowdhury279
Жыл бұрын
Make videos on Cortex
@AntonPutra
Жыл бұрын
in the works with vitoriametrics
@rahulchowdhury279
Жыл бұрын
@@AntonPutra appreciate it. and if possible please make it a detailed and long video
@AntonPutra
Жыл бұрын
@@rahulchowdhury279 sure
@rahulchowdhury279
5 ай бұрын
Still waiting @Anton
@sagarhm2237
11 ай бұрын
Which app u using to edite
@AntonPutra
11 ай бұрын
Adobe Suite
@sagarhm2237
11 ай бұрын
@@AntonPutra hey bro can do one vedio on opentelometry auto instrumentation kubernates cluster which application running on pod and i need response time of endpoints in that application. I'm a beginner and I'm not getting how approach and implementation.
@AntonPutra
11 ай бұрын
@@sagarhm2237 Sure, in the future. I already used it in one of my benchmark videos, but I don't remember which one exactly. However, it is not as mature as Prometheus. On the other hand, it's used by many commercial monitoring agents.
Пікірлер: 150