amazing stuff. thank you so much for that. i was wondering if spark is a must or can we just use Dremio to do the data ingestion too?
@Dremio
6 ай бұрын
Dremio can do a lot of ingestion work, and those capabilities are growing everyday. - Using CTAS, INSERT INTO and COPY INTO commands we can move data from any of our sources into Apache Iceberg tables on our data lake.
@user-gu4eb9xi7j
5 ай бұрын
Awesome.. Just what I was looking to get rid of AWS. How can I create tables from a CSV file uploaded in minio?
@Dremio
5 ай бұрын
This should help -> www.dremio.com/blog/ingesting-data-into-apache-iceberg-tables-with-dremio-a-unified-path-to-iceberg/
@gfinleyg
4 ай бұрын
Is there a new link for the article? The Flink+Nessie article is still available, but the "Blog Tutorial" link is dead.
@Dremio
4 ай бұрын
both links still seem to be working for me.
@aesthetic_mard
2 ай бұрын
we cant able to read files direcly from minio bucket to appache spark . How can we can read file from mino bucket and process in spark ?
@Dremio
2 ай бұрын
If your following this tutorial sometimes Spark has some weird dns issues with the docker network. The solution is to use the ip address of the Nessie container which you can find by inspecting the network in the docker desktop ui or inspecting the network using the docker CLI to find the ip address of the Nessie container. If you run into a "Unknown Host" issue using minio:9000 then there may be an issue with the DNS in your Docker network that watches the name minio with the ip address of the image on the docker network. In this situation replace minio with the containers ip address. You can look up the ip address of the container with docker inspect minio and look for the ip address in the network section and update the STORAGE_URI variable for example STORAGE_URI = "172.18.0.6:9000"
@Dremio
2 ай бұрын
This tutorial does the same thing without spark www.dremio.com/blog/intro-to-dremio-nessie-and-apache-iceberg-on-your-laptop/
@joeingle1745
Ай бұрын
Great article Alex. Slight issue creating a view in Dremio, I get the following exception "Validation of view sql failed. Version context for table nessie.names must be specified using AT SQL syntax". Nothing obvious in the console output, any ideas?
@AlexMercedCoder
Ай бұрын
That means the table is in Nessie and it needs to know which branch your using so it would be AT BRANCH main
@joeingle1745
Ай бұрын
@@AlexMercedCoder Thanks Alex. This would seem to be a limitation of the 'Save as View' dialogue, as it doesn't allow me to do this and it doesn't default to the branch you're in the context of currently.
@joshuajames7231
4 ай бұрын
I got an error Failed to load class "org.slf4j.impl.StaticLoggerBinder", when running the script for spark
@Dremio
4 ай бұрын
I'd have to see the whole log output and catalog settings to determine the issue. If you want message me on LinkedIn and I can examine further. - Alex Merced
@khushimuddi7337
26 күн бұрын
even i am getting the same error
@marceloacarrasco
5 ай бұрын
Awesomw tutorial, just a question, trying to create the table, I'm getting this error (can you help).... { "name": "Py4JJavaError", "message": "An error occurred while calling o64.sql. : java.lang.NoClassDefFoundError: org/apache/spark/sql/catalyst/expressions/AnsiCast \tat org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions.$anonfun$apply$6(IcebergSparkSessionExtensions.scala:54) .....
@Dremio
5 ай бұрын
I’d need to see the code and the error can you send me more details at Alex.merced@dremio.com or provide as much context as you can
Пікірлер: 18