Commit Graph
7 Commits
Author SHA1 Message Date
Rob Mulholand 36c4da372b (VDB-570) Handle duplicate storage diffs
- If processing a new diff for a row that already exists in the DB,
  ignore the error without logging or queueing
- If processing a queued diff for a row that already exists, remove
  it from the queue
2019-05-02 11:39:16 -05:00
Rob Mulholand b036053937 Queue storage diffs if transformer execution fails
- For any error, not just if key isn't recognized
- Means we don't lose track of diffs on random ephemeral errors
2019-05-01 12:32:39 -05:00
Rob Mulholand 6a86de87b4 (VDB-371) Recheck queued storage
- Iterate through queued storage at defined interval, popping rows
  from the queue if successfully persisted
2019-05-01 12:30:37 -05:00
Rob Mulholand 2d684c5aec Extract storage diff fetching behind an interface
- Replaces directly reading from a CSV
- Simplifies testing
- Should hopefully make it easier to plug in other sources for storage
  diffs (e.g. differently formatted CSVs, JSON RPC, etc)
2019-05-01 12:30:36 -05:00
Ian Norden 9f8c50e3ab tests for migration path ordering and errors 2019-03-13 11:42:30 -05:00
Ian Norden 90e67d6da9 move storage mappings pkg into shared 2019-02-24 21:38:48 -06:00
Ian Norden decc2a3caf remove pkg/transformers; extract shared files needed to
libraries/shared; work on automated db migration management
2019-02-24 21:38:47 -06:00