backgroundImage

The fluent way to connect to data sources in GX

Creating a Datasource just got (much) easier

Erin Kapp
April 18, 2023
Example fluent datasource configuration code with decorative linear waves

It’s so much easier to connect to data & create a Datasource in Great Expectations now that we could have fit the whole update in the headline—so here’s a short post for a big announcement.

To illustrate, here’s a little now & then for configuring the same Datasource.

Now: fluent configuration in GX 0.16+

1datasource = context.sources.add_or_update_pandas(name="my_pandas_dataframe")

Then: block configuration in GX before 0.16

1datasource_config = {
2 "name": "my_pandas_datasource",
3 "class_name": "Datasource",
4 "module_name": "great_expectations.datasource",
5 "asset_name": "my_pandas_asset",
6 "execution_engine": {
7     "class_name": "PandasExecutionEngine",
8      "module_name": "great_expectations.execution_engine",
9 },
10 "data_connectors": {
11     "my_pandas_connector": {
12         "module_name": "great_expectations.datasource.data_connector",
13         "class_name": "RuntimeDataConnector",
14     }
15 },
16}
17datasource =  context.add_datasource(**ds_config, save_changes=True)

If you’re not a visual learner, here are some stats. To establish the same Datasource, the examples above take:

  • 77 characters using fluent configuration

  • 534 characters using block configuration

We can’t promise that every fluent configuration will be 86% shorter (or, to be precise, 85.58% shorter) than its block equivalent. But regardless of the exact percentage, the fluent configuration style is a huge upgrade.

Block configuration is still supported, but we encourage everyone to use the fluent configuration for new Datasources and to convert their existing ones when feasible.

For more details about fluent configuration, watch a video demo here or see our docs on connecting to data here

That’s all—happy connecting!

Search our blog for the latest on data quality.


©2024 Great Expectations. All Rights Reserved.