Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix signature saving at Predict saving #1556

Merged
merged 6 commits into from
Oct 8, 2024

Conversation

chenmoneygithub
Copy link
Collaborator

resolve #1045

Change summary:

  • Add a dump_state/load_state pair to Signature class for better modularization.
  • Remove the save_verbose option, as it doesn't make sense to change input fields to output fields at loading time. As we are doing "weights-only" saving, the signatures' architecture should match, i.e., input fields and output fields' number and name. Only changeable attributes should be included with saving.
  • Save prefix for every field as requested in Saving/Loading Method for signatures in dspy.Predict #1045.
  • Added unit testing.


assert CustomSignature2.dump_state() != expected
# Overwrite the state with the state of CustomSignature.
CustomSignature2.load_state(state)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry if I'm forgetting, but is this PR introducing stateful signatures for the first time?

Whether it's old or new, in general, I think it's very risky/hacky to be able to change the object of the signature in-place. Super dangerous IMO. I guess it's fine inside optimizers because they do a deep copy, including of the signatures? In any case, this PR introducing an explicit load_state for signatures seems tricky to me since its usage may not involve a deep copy.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For my education - what part is dangerous? IMO this is only giving user the flexibility to set a signature using saved states, and not affecting existing workflows, or say there shouldn't be any regression.

The original issue was users find that optimizer changes the instruction inside signature, but saving does not capture it so the information gets lost. I don't have the full picture over signature, but I find it's legit to be able to save/load a signature, as it's a component subject to change during optimization.

@okhat okhat merged commit 16ceaba into stanfordnlp:main Oct 8, 2024
4 checks passed
@chenmoneygithub chenmoneygithub deleted the fix-saving branch December 27, 2024 22:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Saving/Loading Method for signatures in dspy.Predict
2 participants