Glue Streaming – Error in SQL statement: AnalysisException: Table or view not found

How to fix the AnalysisException SQL error “Table or view not found”.

Problem:

On a Glue spark Streaming job, the code below fails with error message above

        data_frame.createOrReplaceTempView("footable")
        spark.sql("select * from footable").show()

Explanation:

A streaming query uses its own SparkSession which is cloned from the SparkSession that starts the query. And the DataFrame provided by foreachBatch is created from the streaming query’s SparkSession. Hence you cannot access temp views using the original SparkSession.

Solution:

Create a Global temp view:

        data_frame.createOrReplaceGlobalTempView("footable")
        spark.sql("select * from global_temp.footable").show()

AWS CloudFormation – Using Conditions to determine resource creation

This shows an example of how to create resources based on conditions.
I’ll be using a simple example of creating buckets and applying tags. The “tag” is applied to the bucket that is always created based on whether or not the environment is “prod”. Not a realistic scenario, but a good enough example.

In summary, the stack bellow creates 2 buckets, one of them depending on the “Environment” variable.

AWSTemplateFormatVersion: "2010-09-09"

Parameters: 
  Environment: 
    Description: Environment type.
    Default: test
    Type: String
    AllowedValues: 
      - prod
      - test
    ConstraintDescription: must specify prod or test.

Conditions: 
  CreateProdBucket: !Equals [ !Ref Environment, prod ]

Resources: 
  MyBucket1:
    Type: AWS::S3::Bucket
    Condition: CreateProdBucket
    Properties: 
      BucketName: 'testenvbucket-create-if-prod'

  MyBucket2:
    Type: AWS::S3::Bucket
    Properties: 
      BucketName: 'testenvbucket-always-created'
      Tags:   
        - 'Fn::If':
          - CreateProdBucket
          -
            Key: EnvironmentTag
            Value: CreateProdBucketIsTrue
          -
            Key: EnvironmentTag
            Value: CreateProdBucketIsFalse

If you create the stack with “test”, only one bucket will be created with the expected tag:

Upon updating it to “prod”, you can see it in the “change set preview” that a new bucket will be added, and one tag modified:

BTW, you may need to force a refresh in the S3 console to see the updated tags.