The Glue code that runs on AWS Glue and on Dev Endpoint

-

When you develop code for Glue with the Dev Endpoint, you soon get annoyed with the fact that the code is different in Glue vs on Dev Endpoint
  • glueContext is created in a different manner
  • there's no concept of 'job' on dev endpoint, and therefore
  • no arguments for the job, either
So Mike from The MIS Theorist asked if there was a simpler way. And sure there is!


Template boilerplate Glue code


import sys
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
# Some common stuff not needed in this boilerplate
#from pyspark.sql.functions import *
#from awsglue.dynamicframe import DynamicFrame
#from awsglue.transforms import *

_dev_ep = False
try:
    ## There's no JOB_NAME in args, so code will raise an exception here
    args = getResolvedOptions(sys.argv, ['JOB_NAME', 'DAYS'])
    sc = SparkContext()
    glueContext = GlueContext(sc)
    job = Job(glueContext)
    job.init(args['JOB_NAME'], args)

except Exception as e:
    print("Exception:", e)
    _dev_ep = True
    args = {'JOB_NAME': 'Your Glue Job', 'DAYS': '1'}
    glueContext = GlueContext(SparkContext.getOrCreate())

spark = glueContext.spark_session

## Do your thing after this line
datasource0 = glueContext.create_dynamic_frame.from_catalog(
    database = "your_database_name",
    table_name = "your_table_name")
datasource0.printSchema()

## Don't change the rest
if not _dev_ep:
    job.commit()

This makes developing Glue code easier, since you can copy-paste your development code directly into Glue and it still works.

Let me know if you found this helpful 👇🏻. Cheers! 🙃

Comments

Post a Comment

Popular posts from this blog

How to access AWS S3 with pyspark locally using AWS profiles tutorial

-
At Zervant , we currently use databricks for our ETL processes, and it's quite great. However, there's been some difficulty in setting up scripts that work both locally and on the databricks cloud. Specifically, databricks uses their own prorpietary libraries to connect to AWS S3 based on AWS hadoop 2.7. That version does not support accessing using AWS profiles. Internally, we use SSO to create temporary credentials for an AWS profile that then assumes a role. Therefore, reading the ACCESS_ID and ACCESS_SECRET from the .credentials file is something we don't want to do. In order to accomplish this, we need to set two hadoop configurations to the Spark Context fs.s3a.aws.credentials.provider com.amazonaws.auth.profile.ProfileCredentialsProvider This is done by running this line of code: sc._jsc.hadoopConfiguration().set("fs.s3a.aws.credentials.provider", "com.amazonaws.auth.profile.ProfileCredentialsProvider") Note! You need to set your environment var...
Read more

Snowflake UPSERT operation (aka MERGE)

-
You want to insert data to a table, but if a corresponding row already exists (by some rule, e.g. unique key) you want to update that instead of adding a new row, keeping the dataset's unique requirements intact. That's an "UPDATE AND INSERT" operation, or UPSERT. Some SQL languages have native support for it.  PostgreSQL has UPSERT as native. Also MySQL supports the operation with INSERT and ON DUPLICATE KEY UPDATE. How do you do UPSERT on Snowflake? Here's how: Snowflake UPSERT i.e. MERGE operation Snowflake's UPSERT is called MERGE and it works just as conveniently. It just has a different name. Here's the simple usage: MERGE INTO workspace.destination_table d USING workspace.source_table s ON d.id = s.id AND d.val1 = s.val1 WHEN MATCHED THEN update SET d.val2 = s.val2, d.val3 = s.val3 WHEN NOT MATCHED THEN INSERT (id, val1, val2, val3) VALUES (s.id, s.val1, s.val2, s.val3); Here the destination_table and source_table are of similar form,...
Read more