Glue Streaming – Error in SQL statement: AnalysisException: Table or view not found

How to fix the AnalysisException SQL error “Table or view not found”.

Problem:

On a Glue spark Streaming job, the code below fails with error message above

        data_frame.createOrReplaceTempView("footable")
        spark.sql("select * from footable").show()

Explanation:

A streaming query uses its own SparkSession which is cloned from the SparkSession that starts the query. And the DataFrame provided by foreachBatch is created from the streaming query’s SparkSession. Hence you cannot access temp views using the original SparkSession.

Solution:

Create a Global temp view:

        data_frame.createOrReplaceGlobalTempView("footable")
        spark.sql("select * from global_temp.footable").show()