[FSTORE-1470] Documentation and Guides for Snowflake Schema #398
Conversation
jimdowling
left a comment
jimdowling
left a comment
There was a problem hiding this comment.
Your data model is off.
Check here:
https://docs.google.com/document/d/12slqcD6Fu1DB47CTybAdCBO8NwocZGGK-wovm6mtKdk/edit
merchant_details should be joined on merchant_id
aggregated_cc_trans is a child of cc_transactions.
account_details is a child of aggregated_cc_trans
There was a problem hiding this comment.
In the figures, the PKs should always be the first columns in the tables.
They should be called FK (not PK) in the tables that point to the tables that own the PKs.
There was a problem hiding this comment.
The order of columns in tables should be:
- primary keys
- event_time
- partition keys
- features
- foreign keys
| .join(location_fg.select_all(), left_on=["location_id"], right_on=["id"], join_type="left") | ||
| selected_features = credit_card_transactions_fg.select_all() \ | ||
| .join(account_details_fg.select_all(), on=["cc_num"]) \ | ||
| .join(merchant_details_fg.select_all(), left_on=["cc_num"], right_on=["merchant_id"], join_type="left") |
There was a problem hiding this comment.
Why explicitly set join_type=left?
You should document that's the default join type, but no need to include it here, right?
It's very rare you would need to change the join_type, so KISS here
There was a problem hiding this comment.
This is not a good example:
.join(merchant_details_fg.select_all(), left_on=["cc_num"], right_on=["merchant_id"],
This is, imo, a better example
.join(merchant_details_fg.select_all(), left_on=["merchant_id"], right_on=["id"],
Often the FK has the name "<fg_name>_id", and the FG itself as "id" as its PK.
There was a problem hiding this comment.
Regarding join_type, we are talking about more complex joins. So added comment about default join type and changed to "inner" just to make the point.
| feature_view = fs.create_feature_view( | ||
| name='rain_dataset', | ||
| query=feature_join | ||
| name='credit_card_fraud', |
There was a problem hiding this comment.
Add an explicit version to the feature view?
Labels in the feature view?
|
|
||
| === "Python" | ||
| ```python | ||
| feature_join = rain_fg.select_all() \ |
There was a problem hiding this comment.
Ok, to keep join_type her.e
| ``` | ||
|
|
||
| #### Snowflake schema | ||
| Hopsworks also provides the possibility to define children and grandchildren of the root (the left most) feature group. |
There was a problem hiding this comment.
Hopsworks also provides the possibility to define a feature view that consists of a nested tree of children (to up to a depth of 10) from the root (left most) feature group.
There was a problem hiding this comment.
Do we have a max depth for the tree?
| #### Snowflake schema | ||
| Hopsworks also provides the possibility to define children and grandchildren of the root (the left most) feature group. | ||
| This is called Snowflake Schema data model where you need to build nested tables (subtrees) using joins, and then join the | ||
| subtrees to their parents which can be children of the root node (the leftmost feature group): |
There was a problem hiding this comment.
subtrees to their parents iteratively until you reach the root node (the leftmost feature group in the feature selection)
|
|
||
| selected_features = credit_card_transactions.select_all() | ||
| .join(nested_selection) | ||
| .join(merchant_details.select_all() |
There was a problem hiding this comment.
Missing a closing parentheses "):
| .join(merchant_details.select_all() | ||
| ``` | ||
|
|
||
| Now, you have the benefit that in online inference you only need to pass two foreign key values to retrieve the precomputed features: |
There was a problem hiding this comment.
pass two serving key values (the foreign keys of the leftmost feature group)
| .join(location_fg.select_all(), left_on=["location_id"], right_on=["id"], join_type="left") \ | ||
| .filter((rain_fg.location_id == 10) | (rain_fg.location_id == 20)) | ||
| selected_features = credit_card_transactions_fg.select_all() \ | ||
| .join(account_details_fg.select_all(), on=["cc_num"]) \ |
There was a problem hiding this comment.
its account_details_fg, cc_num is the foreign key here in credit_card_transactions_fg and id or transaction_id will be pk
3 participants
No description provided.